Open-source alternatives to MC Planner

CLIPort: What and Where Pathways for Robotic Manipulation Mohit Shridhar, Lucas Manuelli, Dieter Fox CoRL 2021

[Website](http://craftjarvis-jarvis1.github.io/) [Paper](https://arxiv.org/abs/2311.05997) [Twitter](https://twitter.com/jeasinema/status/1723900032653643796)

This repo contains the source code for making plans based on problems decribed by natural language.

We instantiate Pangu on knowledge base question answering (KBQA), which is representative testbed for grounded language understanding with a highly complex and heterogeneous environment.

This repository contains the code for the paper What Can RL Bring to VLA Generalization? An Empirical Study. The pretrained checkpoints are available at HuggingFace.

Transformers is a comprehensive library for machine learning that provides a unified interface for training, fine-tuning, and deploying transformer-based models. It supports a wide range of tasks, including text classification, language modeling, question answering, and sequence-to-sequence translation, while offering specialized architectures for both text and vision processing. The framework includes tools for managing the entire model lifecycle, from data preprocessing and tokenization to distributed training and inference. The library features extensive support for model optimization and

With Self-Refine, LLMs can generate feedback on their work, use it to improve the output, and repeat this process.

Develop and deploy smart contracts using natural language.

JARVIS is a system for large language model task orchestration, deployment management, and automation benchmarking. It utilizes a task orchestrator to decompose complex requests into actionable steps and coordinates various expert models to synthesize final responses. The project includes an AI model deployment manager to handle the local deployment of expert models across different hardware scales. It further provides an AI workflow API consisting of web endpoints used to trigger automated task workflows and retrieve results from model selection stages. The framework incorporates an automat

UFO is a multi-device task orchestrator and LLM agent orchestration framework designed to decompose natural language requests into executable task graphs. It functions as a cross-platform UI automation tool capable of performing interactions on Windows and mobile devices while routing tasks to distributed agents based on their hardware and software capabilities. The system is distinguished by its RAG-enhanced agent architecture, which integrates external documentation and previous execution traces to improve decision-making. It employs a hybrid UI detection approach that combines computer vis

Voyager is an autonomous embodied agent and lifelong learning framework that uses a large language model to explore virtual environments. It functions as a code-based action controller, translating natural language instructions into executable scripts to interact with its surroundings. The system features an automatic curriculum generator that creates sequences of exploration goals to discover new items and behaviors without human intervention. It maintains a skill library manager that stores learned behaviors as reusable code fragments, which can be composed to execute complex tasks. The fr

[Paper](https://arxiv.org/abs/2210.05663) [Website](https://mahis.life/clip-fields/) [Code](https://github.com/notmahi/clip-fields) [Data](https://osf.io/famgv) [Video](https://youtu.be/bKu7GvRiSQU)

AgiBot-World is a suite of software pipelines and tools designed for robotic policy training, dataset standardization, embodiment transfer, and performance benchmarking. It provides infrastructure for developing bimanual manipulation policies using foundation models and human-reference trajectory data. The project features a robot embodiment transfer suite that adapts pre-trained models to different robot bodies without requiring new multi-embodiment training data. It also includes a specialized evaluation framework for validating vision-language-action models through open-loop testing and ph

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation Mohit Shridhar, Lucas Manuelli, Dieter Fox CoRL 2022

Welcome to the codebase for our paper, "Language Models can Solve Computer Tasks". In this codebase, you will find the implementation of our RCI agent, which uses a pre-trained language model to execute computer tasks in MiniWoB++ benchmark guided by natural language. The agent employs a simple…

Note: https://github.com/kyegomez/tree-of-thoughts CANNOT replicate paper results.