How MediaPipe sees a hand — and exactly where it fails

contact@mulhamfetna.com (Mulham Fetna) — Tue, 08 Sep 2026 00:00:00 +0000

Twenty-one points, thirty frames a second, on a CPU, from a flat RGB image with no depth information. This is how that is possible — and the four failure modes you will meet the first time you rely on it.

Every vision-driven robotics project has a moment where the camera stops being a camera and starts being a sensor. For this one, that moment is MediaPipe Hands: a webcam frame goes in, and 21 numbered points in space come out.

It is easy to treat that as a black box. It is also a mistake, because the box has a specific shape, and its failure modes follow directly from how it was built.

It is two networks, not one
#

The single most useful thing to know about MediaPipe Hands is that it is a cascade: a detector and a regressor, with completely different jobs.

flowchart TB
    A["📷 Full frame
e.g. 640 × 480"] --> B["Stage 1 — BlazePalm
SSD detector"]
    B --> C["Oriented palm crop
256 × 256"]
    C --> D["Stage 2 — Landmark regressor
MobileNetV2-style encoder"]
    D --> E["63 floats
21 landmarks × (x, y, z)"]
    D --> F["Presence score"]
    D --> G["Handedness
left / right"]
    F -->|"confidence ≥ 0.5"| C
    F -->|"confidence < 0.5"| B

The regressor’s output, drawn back onto the frame: 21 points and the connections between them.

Stage 1 — BlazePalm detects palms, never fingers
#

This is the design decision the whole system rests on.

Machine-Learning on Mulham Fetna

How MediaPipe sees a hand — and exactly where it fails

It is two networks, not one #

Stage 1 — BlazePalm detects palms, never fingers #

It is two networks, not one
#

Stage 1 — BlazePalm detects palms, never fingers
#