Examples / Underactuated control

Underactuated control

Underactuated systems are mechanical systems that cannot be fully driven by their controls, such as a two-link arm with a motor only at its elbow. The agent controls the torques of the driven joints to bring the whole system to its goal.

Swing up and stay upright

Pendulum-v1
0.00 s

Principles

Key quantity
angular velocity \(\omega\)
Threshold speed
\(\omega^*(\theta)=\sqrt{\tfrac{3g}{l}\,(1-\cos\theta)}\), the speed that just coasts to the top, taken at the next-step angle \(\theta'\)
PrincipleWhenReference
Accumulate energy\(|\theta|>0.41\) rad\(\omega\to\operatorname{sgn}(\omega)\,\omega^*(\theta')\)
Stay upright\(|\theta|\le 0.41\) rad\(\omega\to-\operatorname{sgn}(\theta')\,\omega^*(\theta')\)

\(\theta\) is measured from upright and \(\omega\) is its rate. At 0.41 rad the largest torque just holds the pendulum against gravity.

Sample efficiency

Within 36,000 environment steps

Success criterion
all 20 evaluation episodes stay within 0.1 rad of upright for their last 2.5 s
Converges in
18 updates

Train it yourself

pendulum_swing_up.py requirements.txt 
$ pip install -r requirements.txt
$ python pendulum_swing_up.py

Swing the tip above the line

Acrobot-v1
0.00 s

Principles

Key quantity
elbow angular velocity \(\omega_2\), the rotation of the lower link relative to the upper link
PrincipleWhenReference
Accumulate energyalways\(\omega_2\to4\pi\operatorname{sgn}(\omega_2)\) rad/s

Pushing along the elbow rotation makes the motor add energy at every step.

Sample efficiency

Within 2,000 environment steps

Success criterion
all 20 evaluation episodes lift the tip above the line within 500 steps
Converges in
1 update

Train it yourself

acrobot_swing_up.py requirements.txt 
$ pip install -r requirements.txt
$ python acrobot_swing_up.py

Keep the pole upright

CartPole-v1
0.00 s

Principles

Key quantity
pole angular velocity \(\omega\)
Return speed
\(\omega^*(\theta)=\sqrt{2a\,|\theta|}\), with \(a\) a quarter of the angular acceleration the push gives the pole, taken at the next-step angle \(\theta'\)
PrincipleWhenReference
Stay uprightalways\(\omega\to-\operatorname{sgn}(\theta')\,\omega^*(\theta')\)

\(\theta\) is measured from upright and \(\omega\) is its rate. The pole turns back fast when far from upright and slows down as it arrives.

Sample efficiency

Within 2,000 environment steps

Success criterion
all 20 evaluation episodes keep the pole up and the cart on the track for 500 steps
Converges in
1 update

Train it yourself

cartpole_balance.py requirements.txt 
$ pip install -r requirements.txt
$ python cartpole_balance.py