MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

Where MediaPipe sits

#

flowchart TB
    CAM["USB webcam
640×480 @ 30 fps · YUYV"] -->|"BGR frame"| RGB["cv2.cvtColor
BGR → RGB"]
    RGB --> MP["MediaPipe Hands
palm detector + landmark model"]
    MP -->|"21 × (x, y, z)"| ANG["Triplet angles
Part 5"]
    ANG --> AVG["Average per finger
Part 6"]
    AVG --> NORM["Flexion 0..1
Parts 7–8"]
    NORM -->|"/hand/target_flexions"| TWIN["MuJoCo twin"]

The vision layer knows nothing about MuJoCo. Its entire output is five numbers between 0.0 (open) and 1.0 (closed).

Two networks, not one

#

MediaPipe Hands is a cascade:

1. BlazePalm, a single-shot detector, finds a palm bounding box in the full frame. Palms, not hands: a palm is close to a rigid square; a hand with moving fingers is not.
2. A landmark model crops that region and regresses 21 keypoints, a hand-presence score and handedness.

In video mode (static_image_mode=False) the detector barely runs. Landmarks from frame t define the crop for frame t+1, and the detector wakes only when tracking confidence drops. That shortcut is why this pipeline holds 30 fps on a CPU.

The 21 landmarks

#

What it looks like live

#

Previous Next

Each frame: the tracker window with draw_landmarks output, the twin, and the tendon forces the resulting flexions produced. Notice the fist — MediaPipe still resolves all five fingers with the hand foreshortened toward the camera.

The trap: what the coordinates actually are

#

Each landmark in results.multi_hand_landmarks has x, y and z:

• x and y are normalized by image width and height separately — x = pixel_x / 640, y = pixel_y / 480.
• z is relative depth with the wrist as origin, roughly on the scale of x; smaller is closer. It is inferred from one RGB image, not measured.

The code builds vectors from (pt.x, pt.y, pt.z) and measures angles between them. Dividing x and y by different numbers stretches the space, so the angle you measure depends on how the finger is oriented in the frame.

We checked it numerically — a true 90° bend drawn in a 640×480 frame:

A 16° error from orientation alone. The empirical calibration constants (Part 7) absorb most of it for a hand held upright — which is why the pipeline works — but tilt your hand and the thresholds drift.

The fix: world landmarks

#

MediaPipe also returns results.multi_hand_world_landmarks: the same 21 points in metres, in a hand-centred 3D frame with no image aspect ratio involved. Angles between those vectors don’t care how the hand is oriented or how far it is from the camera.

# today: image-normalized, aspect-ratio dependent
lm = np.array([(pt.x, pt.y, pt.z) for pt in results.multi_hand_landmarks[0].landmark])

# production-grade: metric and orientation-invariant — recalibrate after switching
lm = np.array([(pt.x, pt.y, pt.z) for pt in results.multi_hand_world_landmarks[0].landmark])

It is a one-line change plus a recalibration, and the first item on the production roadmap.

Frame handling, line by line

#

frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)   # OpenCV captures BGR; the models expect RGB
frame_rgb.flags.writeable = False                     # lets MediaPipe use the buffer without copying
results = self.hands.process(frame_rgb)
frame_rgb.flags.writeable = True

• BGR → RGB is mandatory. BGR still “works” but degrades detection.
• writeable = False is MediaPipe’s documented hint to avoid a full-frame copy.
• No hand → every flexion is 0.0, so the twin opens when you leave the frame — a deliberate fail-open default that a robot holding an object would want to change.

What it costs

#

On the development laptop (Intel Comet Lake, 12 threads), 640×480 frames, mean of 150 frames:

Identical in the container and on the host. Why it gets three to four times slower when the MuJoCo viewer is open is Part 19.