Prescriptive Policy Learning
for pendulums

PPL teaches agents principles of behavior, such as how to maintain balance, how to accumulate energy, and how to land smoothly. Compared with rewards, PPL enables agents to learn good, generalizable policies directly from principles of behavior.

Teaching Agents the “Principle”, Not the “Reward”: Prescriptive Policy Learning
Shan Yang, National University of Singapore

Examples

Each example shows the behavior the agent learns, the principles it is taught, and the code to train it.

Underactuated control

Underactuated systems are mechanical systems that cannot be fully driven by their controls, such as a two-link arm with a motor only at its elbow. The agent uses the coupling between driven and undriven parts to reach a target configuration or maintain balance.

Thrust-driven vehicles

Thrust-driven vehicles move only through the forces of their engines, such as a lander with a main engine and two side engines. The agent coordinates the vehicle's translation and rotation to reach a target position or follow a desired trajectory.

More examples in progress

We are adding further tasks from Gymnasium and beyond.

Mountain CarReacherQuadrotorCar Racing HopperWalker2dBipedal WalkerAnt Humanoid StandupPusherFetch PushAdroit Hand

Why is learning from reward hard?

Task with wind wind land softly on the pad Reward per step + closer + slower + upright +10 leg contact −0.3 main engine −0.03 side engine +100 at rest −100 crash One landing attempt t = 0 +0.9 −0.2 +1.1 −0.6 +0.4 −1.3 −100 crash ? this action? ? several steps? ? the wind?
Learning from the reward in Lunar Lander with wind.

Reward decoding burden

One step the actionthe state it starts fromthe wind one reward
One landing hundreds of actionsgusts one final reward
The agent ?decodes from these scalars which action it should change

Principles in tasks

Pendulum

Build up energy, then keep balance

Lunar Lander

Land softly and stay upright

Mountain Car

Build up energy to reach the top

Cart-pole

Keep the pole over the cart

Reacher

Move toward the goal posture

Hopper

Push off, then land softly

Principles of behavior

A principle is a simple norm of good behavior. It says how to behave, not which action to take, so the same principle holds when the state or the dynamics change.

  • Keep your balancekeep the body over its support
  • Build up energypush along the motion to swing higher
  • Slow down before arrivingbrake in time to stop where you must
  • Keep clear of obstaclesstay as far from them as you can
  • Land softlytouch down slowly and absorb the impact
  • Move toward the goal postureturn each joint toward where it should end

Click the examples above to see how the principles of each task are taught and what the agent learns.

Implementation of PPL

From principle to reference closer upright slower velocity now reference references computed from height, tilt and position Each action is scored how far one step moves the velocity toward its reference −0.3 no engine reference now +0.6 main engine +0.2 side engine −0.2 other side Update actions with a higher score become more likely early trained
A principle guides every action in Lunar Lander.
1

Principle

Stay upright, approach the pad, and do not descend too fast.

2

Key quantity

What the action changes directly in one step, here the velocities.

3

Reference

The velocity to reach after this step, from the height, tilt and position.

4

Each step

An action scores by how far it moves the velocity toward the reference.

Score each action

\[ \delta(s,a)=\tfrac12\big\|\widetilde{\bm q}(s)-\bm q(s)\big\|^2-\tfrac12\big\|\widetilde{\bm q}(s)-\bm q(s')\big\|^2 \]

How much closer one step brings the key quantity to its reference.

Make good actions more likely

\[ \theta\leftarrow\theta+\eta\,\mathbb{E}\big[\delta(s,a)\,\nabla_\theta\log\pi_\theta(a\mid s)\big] \]

Actions with a higher score become more likely.

\(\bm q\) key quantity, such as an angular velocity · \(\widetilde{\bm q}(s)\) the value it should reach after this step · \(s, s'\) the state before and after the action · details in the paper ↗