Examples / Thrust-driven vehicles

Thrust-driven vehicles

Thrust-driven vehicles move only through the forces of their engines, such as a lander with a main engine and two side engines. The agent decides which engine to fire at each moment to steer the vehicle where it should go.

Stabilize above the pad

LunarLander-v3
0.00 s

Principles

Key quantities
vertical velocity \(v_y\), angular velocity \(\omega\), horizontal velocity \(v_x\)
Approach speed
\(v^*(e,a)=\operatorname{sgn}(e)\min\big(\sqrt{2a|e|},\,|e|/\tau\big)\): fast when far, slowing down on arrival
Target tilt
\(\tilde\theta=\theta_l\operatorname{clip}\big((v_x-v^*(-x',a_x))/0.5,\,-1,\,1\big)\), the lean the remaining \(v_x\) still needs
PrincipleWhenReference
Return to the heightalways\(v_y\to v^*(h_{\text{target}}-h',\,a_y)\)
Return to the centrealways\(v_x\to v^*(-x',\,a_x)\)
Lean as neededalways\(\omega\to\operatorname{sgn}(\tilde\theta-\theta')\sqrt{2a_\omega|\tilde\theta-\theta'|}\)

Primes mark the next state with the engines off; \(h\) is the height above the pad and \(h_{\text{target}}\) the target height, 7 m.

Sample efficiency

Within 33,767 environment steps

Success criterion
all 20 evaluation episodes stay between the flags within 0.5 m of \(h_{\text{target}}\) for their last 5 s
Converges in
15 updates

Train it yourself

lunar_lander_stabilize.py requirements.txt 
$ pip install -r requirements.txt
$ python lunar_lander_stabilize.py

Land from the top

LunarLander-v3
0.00 s

Principles

Key quantities
vertical velocity \(v_y\), angular velocity \(\omega\), horizontal velocity \(v_x\)
Target tilt
\(\tilde\theta=\theta_l\,k\operatorname{clip}\big((v_x-v^*(-x',a_x))/0.5,\,-1,\,1\big)\), with \(k\) falling to 0 at touchdown
PrincipleWhenReference
Descend safelyin flight\(v_y\to-\sqrt{v_{\text{safe}}^2+2a_yh'}\)
Return to the centrein flight\(v_x\to v^*(-x',\,a_x)\)
Lean as neededin flight\(\omega\to\operatorname{sgn}(\tilde\theta-\theta')\sqrt{2a_\omega|\tilde\theta-\theta'|}\)
Settlea leg toucheseach velocity \(\to\) its next value with the engines off

As in the task above, with \(a_y\) halved. Episodes start at the top of the view with a random push.

Sample efficiency

Within 17,876 environment steps

Success criterion
all 20 evaluation episodes come to rest between the flags
Converges in
25 updates

Train it yourself

lunar_lander_land_from_top.py requirements.txt 
$ pip install -r requirements.txt
$ python lunar_lander_land_from_top.py

Take off and hover

HoverAviary
0.00 s

Principles

Key quantities
vertical velocity \(v_z\), angular velocities \(\omega_x,\omega_y\)
Approach speed
\(v^*(e,a)=\operatorname{sgn}(e)\min\big(\sqrt{2a|e|},\,|e|/\tau\big)\)
Target tilt
\(\tilde n=\sin\theta_l\operatorname{clip}\big((v^*(\rho',a_h)\,u-v_h)/v_c\big)\), the lean the remaining horizontal velocity still needs
PrincipleWhenReference
Climb to the targetalways\(v_z\to v^*(z_{\text{target}}-z',\,a_z)\)
Lean as neededalways\((\omega_y,-\omega_x)\to\frac{\tilde n-n'}{|\tilde n-n'|}\sqrt{2a_\omega|\tilde n-n'|}\)

Primes mark the next state without action; \(n\) is the horizontal part of the thrust axis, \(\rho\) the horizontal distance to the target and \(u\) its direction.

Sample efficiency

Within 59,892 environment steps

Success criterion
flies the full 8 s and stays within 0.1 m of the target for the last 2 s
Converges in
204 updates

Claude Opus 5.5 (medium effort) designed these principles directly from the Lunar Lander experience, without human intervention. A small network with two hidden layers of 64 units learns them within 27 s of training.

Train it yourself

quadrotor_hover.py requirements_quadrotor.txt 
$ pip install -r requirements_quadrotor.txt
$ python quadrotor_hover.py --seed 3