Embodied.cpp is an inference runtime for embodied AI models: Vision-Language-Action (VLA) models and World-Action Models (WAMs) for robotic perception and control. It runs these models efficiently on heterogeneous hardware (CPU / CUDA GPU / NPU) using GGUF weights, and ships with ready-to-use servers and evaluation clients.
- [2026.08] 🔥🔥 Released Embodied.cpp v1.0.
- [2026.07] Added support for Cosmos3-Nano and GR00T N1.7, the RoboLab benchmark, and Isaac Sim.
- [2026.06] Released the initial version of Embodied.cpp with support for pi0.5, HY-VLA, and LingBot-VA, plus the LIBERO and RoboTwin benchmarks.
groot-n1.7-demo-results.mp4
- Successful execution ✅
- Cumulative inference time 2.5 s → 1.7 s (⬇32%).
- End-to-end execution time 14.6 s → 13.3 s.
hy-vla-demo-results.mp4
- Successful execution ✅
- Cumulative inference time 9.4 s → 6.6 s (⬇30%).
- End-to-end execution time 14.8 s → 12.0 s.
- Embodied.cpp 🤖
 pi0.5 |
 SmolVLA |
 HY-VLA |
 GR00T N1.7 |
 LingBot-VA |
 Cosmos3-Nano |
We continuously track advances in embodied AI and adapt Embodied.cpp to the latest open models. Pull requests that add support for new models are always welcome.
VLA results are normalized to each model's Python baseline (1.00), and each cell reports Python → C++. Lower inference latency and VRAM are better; higher success rate is better. C++ denotes the BF16 implementation.
| Model | Inference Latency ↓ | VRAM ↓ |
|---|---|---|
| pi0.5 | 1.00 → 0.90 (10% lower) | 1.00 → 0.60 (40% lower) |
| GR00T N1.7 | 1.00 → 0.72 (28% lower) | 1.00 → 0.93 (7% lower) |
| HY-VLA | 1.00 → 0.48 (52% lower) | 1.00 → 0.68 (32% lower) |
For World Models, C++ substantially reduces VRAM while keeping the success rate close to the Python baseline.
| Model | VRAM ↓ |
|---|---|
| Cosmos3 | 21.84 GB → 19.49 GB (10.8% lower) |
| LingBot-VA | 24.75 GB → 16.44 GB (33.6% lower) |
Highlights: Compared with Python, C++ BF16 reduces VLA inference latency by up to 52% and VRAM by up to 40%. For World Models, it reduces VRAM by up to 33.6%, with success-rate changes limited to 2 percentage points.
- This project is still under active construction 🚧
- A more modular and maintainable runtime architecture for
Embodied.cpp - Additional inference optimizations, such as real-time chunking and VLA caching
git clone <repo-url> && cd embodied.cpp
./patches/init_third_party.shBy default, the setup script prepares a combined llama.cpp source tree for all
supported runtimes. For smaller model-specific setups or custom patch profiles,
see patches/PATCH.md.
Pre-converted GGUF releases for Embodied.cpp are available on Hugging Face:
The repository currently hosts GGUF artifacts prepared for the current
Embodied.cpp runtime, including:
pi0.5: main policy GGUF plus multimodal projector GGUFGR00T N1.7: truncated Qwen3-VL text GGUF, vision projector GGUF, and action-head GGUFHY-VLA-0.5: combined VLA GGUF for RoboTwin and related runtime pathsLingBot-VA: transformer GGUF and companion artifacts used by the LingBot pathCosmos3-Nano: RoboLab WAM GGUF with the Wan VAE encoderSmolVLA: LeRobot policy GGUF plus SigLIP mmproj GGUF
Recommended local layout:
checkpoints/
pi05/
pi05.gguf
pi05-mmproj.gguf
groot-n1/
qwen3vl-backbone-bf16.gguf
qwen3vl-mmproj-bf16.gguf
groot-n1.7-libero-object-action-head-bf16.gguf
Hy-Embodied-0.5-VLA-RoboTwin/
Hy-Embodied-0.5-VLA-RoboTwin_bf16.gguf
Hy-Embodied-0.5-VLA-RoboTwin_q4_K.gguf
lingbot_va/
lingbot_transformer.gguf
...
cosmos3/
cosmos3_robolab_full_w8_with_vae_encoder.gguf
smolvla/
smolvla.gguf
mmproj-smolvla.gguf
You can also convert upstream checkpoints yourself with the scripts in
scripts/, but for most users the Hugging Face GGUF releases are
the fastest way to get started.
Install the required system packages for your platform before building.
Minimum build requirements:
- CMake >= 3.22
- A C++17 compiler, such as GCC 11+ or Clang 14+
- CUDA 12.x, optional and required only for GPU builds
Linux:
Make sure cmake, protobuf=3.20.3, zeromq, cppzmq, pkg-config and uv are
available before building. A typical Ubuntu/Debian native-Linux installation is:
sudo apt-get update
sudo apt-get install -y \
build-essential cmake pkg-config protobuf-compiler libprotobuf-dev \
libzmq3-dev cppzmq-dev libegl1-mesa-dev libglu1-mesa-dev \
libgl1-mesa-dev ffmpeg iproute2CUDA runs additionally require an NVIDIA driver, a compatible CUDA toolkit,
and nvidia-smi. Install uv separately if your distribution does not package
it.
Model switches default to OFF. Enable only the runtimes you need.
CUDA GPU template:
CUDA_HOME="${CUDA_HOME:-$(dirname "$(dirname "$(command -v nvcc)")")}"
CUDA_ARCH=${CUDA_ARCH:-native}
cmake -S . -B <BUILD_DIR> \
-DCMAKE_BUILD_TYPE=Release \
-D<MODEL_BUILD_FLAG>=ON \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_COMPILER="${CUDA_HOME}/bin/nvcc" \
-DCMAKE_CUDA_ARCHITECTURES="${CUDA_ARCH}"
cmake --build <BUILD_DIR> --target <SERVER_TARGET> -j$(nproc)Use a separate <BUILD_DIR> for each model or CMake configuration, such as
build-groot-cuda or build-lingbot-cuda; a build directory stores one CMake
configuration and its generated artifacts.
Replace the placeholders with the model you want to build:
| Model | <MODEL_BUILD_FLAG> |
<SERVER_TARGET> |
|---|---|---|
| pi0.5 | MODEL_BUILD_VLA_PI05 |
vla-server |
| SmolVLA | MODEL_BUILD_VLA_SMOLVLA |
vla-server |
| HY-VLA | MODEL_BUILD_VLA_HY_VLA |
vla-server |
| GR00T N1.7 | MODEL_BUILD_VLA_GROOT_N1 |
vla-server |
| LingBot-VA | MODEL_BUILD_WAM_LINGBOT_VA |
wam-lingbot-server |
| Cosmos3-Nano | MODEL_BUILD_WAM_COSMOS3 |
wam-server |
CUDA_ARCH defaults to native, so CMake detects the GPU installed on the
build machine. Override it with an explicit architecture when cross-compiling or
when using CMake older than 3.24. Common explicit values include 75 (Turing),
80 or 86 (Ampere), 89 (Ada), 90 (Hopper), and 120 (Blackwell). The
selected CUDA toolkit must support that architecture; for example, Blackwell
sm_120 requires CUDA 12.8 or newer.
./<BUILD_DIR>/<SERVER_TARGET> <MODEL_ARGUMENTS>Use the <BUILD_DIR> and <SERVER_TARGET> selected in section 2.4. Replace
<MODEL_ARGUMENTS> with the arguments for the selected model:
| Model | <SERVER_TARGET> |
<MODEL_ARGUMENTS> |
|---|---|---|
| pi0.5 | vla-server |
<MMPROJ_GGUF> <MODEL_GGUF> |
| SmolVLA | vla-server |
<MMPROJ_GGUF> <MODEL_GGUF> |
| HY-VLA | vla-server |
<MODEL_GGUF> |
| GR00T N1.7 | vla-server |
--backbone <BACKBONE_GGUF> <MMPROJ_GGUF> <ACTION_HEAD_GGUF> |
| LingBot-VA | wam-lingbot-server |
<TRANSFORMER_GGUF> <TEXT_ENCODER_GGUF> <VAE_ENCODER_GGUF> |
| Cosmos3-Nano | wam-server |
<MODEL_GGUF> |
VLA servers bind to tcp://*:5555 by default. LingBot-VA and Cosmos3-Nano
bind to tcp://*:5557 by default. Pass --bind <ADDR> to override the
listening address or port.
Start the required server as described in section 2.5, then select the configuration and runner for the model and benchmark you want to evaluate.
| Model | Benchmark | Configuration | Server |
|---|---|---|---|
| pi0.5 | LIBERO | pi0.5 | Manual |
| SmolVLA | LIBERO | SmolVLA | Manual |
| GR00T N1.7 | LIBERO | GR00T | Manual |
| LingBot-VA | LIBERO | LingBot | Manual |
| HY-VLA | RoboTwin | HY-VLA | Managed |
| Cosmos3-Nano | RoboLab | Cosmos3 | Managed |
LIBERO uses eval/client/run_sim_client_direct.py; start its matching server
separately. RoboTwin and RoboLab runners start their servers from the selected
configuration and stop them when the evaluation finishes.
SmolVLA on LIBERO:
Convert a LeRobot LIBERO checkpoint into the policy GGUF and the SigLIP
identity-proxy mmproj. The pixel-shuffle connector is stored in the policy
GGUF and executed by models/smolvla.cpp.
python scripts/convert_smolvla_to_gguf.py \
--ckpt checkpoints/smolvla_libero \
--out checkpoints/smolvla/smolvla.gguf
python scripts/convert_smolvla_mmproj_to_gguf.py \
--ckpt checkpoints/smolvla_libero \
--out checkpoints/smolvla/mmproj-smolvla.gguf
# Start `vla-server` with MODEL_BUILD_VLA_SMOLVLA=ON, then run the smoke test.
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl \
eval/sim/libero/libero_uv/.venv/bin/python \
eval/client/run_sim_client_direct.py \
--conf eval/conf/libero_smolvla_eval.yamlThe serialized SmolVLA processor requires a trailing newline in each task prompt; the direct client applies it automatically. The checked-in configuration uses one replayed action per model request and the full LIBERO episode horizon.
See eval/SMOLVLA_VALIDATION.md for the acceptance matrix, smoke protocol, build matrix, and parity methodology. The implementation and validation report is available at eval/SMOLVLA_TECHNICAL_REPORT_ZH.md.
HY-VLA on RoboTwin:
Each checked-in YAML is a baseline evaluation configuration. Adjust its task selection, episode count, model paths, output location, and other benchmark-specific settings for your run; the exact field names are documented in the corresponding configuration and simulator README.
LIBERO tests robotic manipulation skills on the spatial, object, goal,
short, and long suites. Install the simulator once:
bash eval/sim/libero/setup_libero.shAfter starting the matching server in another terminal, run a checked-in configuration:
# pi0.5
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
--conf eval/conf/libero_pi05_eval.yaml
# GR00T N1.7
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
--conf eval/conf/libero_groot_n1_eval.yaml
# LingBot-VA
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
--conf eval/conf/libero_lingbot_va_eval.yamlSee eval/sim/libero/README.md for LIBERO suite
selection, headless EGL execution, and configuration details.
RoboTwin is a dual-arm manipulation benchmark. Install it once:
bash eval/sim/robotwin/setup_robotwin.shRun HY-VLA with the standard configuration:
eval/sim/robotwin/robotwin_uv/.venv/bin/python \
eval/client/run_robotwin_eval.py \
--conf eval/conf/robotwin_hy_vla_eval.yamlSee eval/sim/robotwin/README.md for detailed setup modes and troubleshooting.
RoboLab evaluates the native C++ Cosmos3 WAM path. Install RoboLab once without launching its optional Isaac Sim smoke test:
RUN_SMOKE_TEST=0 bash eval/sim/robolab/setup_robolab.shRun Cosmos3-Nano with the standard configuration:
python3 eval/client/run_robolab_eval.py \
--conf eval/conf/robolab_cosmos3_eval.yamlThe runner starts the C++ wam-server automatically. See
eval/sim/robolab/README.md for configuration
details, the transport-only smoke test, and the PyTorch-reference path.
Pre-converted GGUF releases are available on Hugging Face. Use the conversion tools only when preparing a compatible upstream checkpoint or a custom quantization.
| Model | Workflow |
|---|---|
| pi0.5 | Policy and vision projector |
| GR00T N1.7 | Action head and Qwen3-VL backbone |
| HY-VLA | Combined GGUF and quantization |
| LingBot-VA | Model artifacts and Wan quantization |
| Cosmos3-Nano | RoboLab full_w8 GGUF |
See scripts/README.md for prerequisites, commands,
expected outputs, and post-conversion checks.
What lives where, in plain language:
| Directory | What it contains |
|---|---|
models/ |
C++ implementations of supported models |
runtime/ |
Model registry, architecture detection, shared utilities |
adapter/ |
Typed I/O boundary between observations and model inputs |
serving/ |
ZeroMQ/Protobuf inference servers and API definitions |
kernels/ |
Custom CUDA kernels for GPU builds |
scripts/ |
GGUF conversion and quantization tools |
patches/ |
Third-party setup patches |
eval/ |
Evaluation clients, configurations, and simulator integrations |
If you find Embodied.cpp useful in your research, please consider citing:
@article{xu2026embodiedcpp,
title={Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots},
author={Xu, Ling and Han, Chuyu and Li, Borui and Wu, Hao and Jiang, Shiqi and Cao, Ting and Li, Chuanyou and Zhong, Sheng and Wang, Shuai},
journal={arXiv preprint arXiv:2607.02501},
year={2026},
doi={10.48550/arXiv.2607.02501},
url={https://arxiv.org/abs/2607.02501}
}This project is released under the Apache License 2.0. Third-party dependencies, model checkpoints, datasets, and upstream reference implementations are distributed under their own licenses.
Supported models:
Foundational projects this build depends on: