Diffusion policy • Humanoid robotics

ADAPT Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control

A reusable diffusion prior for real-time language-driven humanoid control.

Yan Wu, Chenhao Li, Kaifeng Zhao*, Gen Li*, Marco Hutter, Siyu Tang
ETH Zurich, Switzerland
*Equal contribution
arXiv Paper Video Code (Coming Soon)
50 Hz End-to-end closed-loop control
Real-time Policy inference
Prompt Motion Online prompt switching
G1 Real-world deployment
01 — Overview

A controller that moves at the speed of language

ADAPT maps changing text commands directly to dynamically feasible whole-body actions.


ADAPT teaser.

ADAPT is an end-to-end diffusion-based skill prior that enables interactive text-driven humanoid control with smooth and robust skill transitions, and can be reused for downstream tasks to achieve goal reaching with controllable motion styles.

In brief

Abstract

We present ADAPT, an end-to-end framework for interactive, text-conditioned humanoid whole-body control. Unlike dominant text-to-motion pipelines that generate kinematic motions for a separate tracker, ADAPT solves language control with an end-to-end closed-loop control framework, where the robot must continuously respond to changing commands while maintaining balance, natural motion, and smooth transitions. ADAPT learns a diffusion-based action prior from text-labeled humanoid state-action trajectories, enabling diverse motion skills to be directly executed from language commands. To improve long-horizon robustness and smooth prompt switching, we train a lightweight residual reinforcement learning policy on top of the frozen diffusion controller. We further show that the same diffusion policy can be reused as a steerable text-conditioned motion prior for downstream task adaptation. Experiments demonstrate robust language-grounded skill execution, smooth interactive transitions, and style-preserving downstream control.

02 — Method

Close the loop between text and motion

One reusable diffusion prior supports interactive control, robust deployment, and downstream adaptation.

01

Interactive control is a dynamic control problem

Interactive language-conditioned humanoid control is a closed-loop dynamic control problem, not a motion generation problem. We therefore train a text-conditioned diffusion policy that maps proprioceptive history + text straight to joint-level actions, together with the future states those actions are expected to produce.

Because supervision comes from frame-level BABEL annotations rather than clip-level captions, the policy is trained on precise text-to-motion grounding and on the transition boundaries between behaviors — exactly the moments an interactive controller has to handle.

Challenge 01

Kinematics–dynamics mismatch

Two-stage pipelines can generate dynamically infeasible references during rapid command switches.

ADAPT

Diffusion skill prior

An 8-layer causal transformer denoiser predicts joint future actions and proprioceptive states, conditioned on a frozen CLIP text embedding. Per-frame independent noise levels match the autoregressive rollout at inference. Using two DDIM steps, the policy supports 50 Hz closed-loop deployment.

Challenge 02

Coarse temporal supervision

Sequence-level captions hide intra-sequence command changes and transition boundaries.

ADAPT

Frame-level grounding

Frame-level BABEL annotations precisely align language with behavior and expose the transitions an interactive controller must handle. The policy learns both sustained skills and command switches instead of treating every clip as a single action.

Challenge 03

Long-horizon distribution shift

Offline behavior cloning drifts outside its training support at arbitrary switch times, causing falls.

ADAPT

Constrained residual correction

A lightweight PPO residual policy corrects the frozen prior. It acts on the lower body only, its scale is warmed up from zero, and a self-tracking reward keeps the corrected state near the diffusion-predicted next state. Prompts switch randomly every 5–10 s during training.

Extension 04

Task adaptation without collapse

A downstream objective must guide behavior without erasing the learned motion semantics.

ADAPT

Noise steering

The frozen prior is reused by learning a policy that outputs the diffusion input noise conditioned on the task goal. Steering biases motion toward the objective while staying inside the learned skill prior.

Method overview.

Method overview. We pretrain a text-conditioned diffusion skill prior on retargeted humanoid motion data (A, B), then adapt it via two complementary mechanisms: residual RL corrects the prior's actions to improve robustness for interactive control (C), while noise steering RL guides the frozen prior toward task goals with controllable motion styles (D).

03 — Results

One policy, many ways to move

From individual skills to long-horizon prompt switching, ADAPT stays responsive and balanced.

Text-driven atomic skills

A single policy executes diverse locomotion, exercise, and upper-body skills from language, all deployed on a Unitree G1.


Interactive prompt switching

Text commands change on the fly, and ADAPT transitions between skills without losing balance.

Long-horizon rollout in simulation with continuous command switching

Long-horizon rollout in the real world with continuous command switching

More skill transitions results.


Goal reaching with motion styles

The same frozen diffusion prior is steered through its input noise to reach a target location while preserving the text-specified style — run, walk while bending over, or the unseen prompt jog.

“run” to goal (green tape)

“walk while bending over” to goal (green tape)


04 — Cite

BibTeX

@article{wu2026adapt,
  title={ADAPT: Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control},
  author={Wu, Yan and Li, Chenhao and Zhao, Kaifeng and Li, Gen and Hutter, Marco and Tang, Siyu},
  journal={arXiv preprint arXiv:2609.00677},
  year={2026}
}

If you find this work useful, you may also be interested in our related work UniPhys and NaP-Control:

@inproceedings{wu2025uniphys,
  title={UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control},
  author={Wu, Yan and Karunratanakul, Korrawe and Luo, Zhengyi and Tang, Siyu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2025}
}

@inproceedings{chen2026napcontrol,
  title={NaP-Control: Navigating Diffusion Prior for Versatile and Fast Character Control},
  author={Chen, Chia-Wen and Wu, Yan and Karunratanakul, Korrawe and Tang, Siyu},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2026}
}