Python on Mulham Fetna

Finger joint angles from three hand landmarks: the dot-product geometry

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Every knuckle angle in this project comes from three landmarks and one dot product. No learning, no lookup table — just the definition of the angle between two vectors, plus two guards that keep a single glitchy frame from sending NaN into a motor command.

Three points make an angle
#

An angle needs a vertex and two rays. A knuckle is the vertex; the two bones meeting there are the rays:

a base point, where the previous bone starts,
the vertex — the knuckle being measured,
an end point, where the next bone ends.

flowchart LR
    P1(("p1
base")) -- "v1 = p1 − p2" --- P2(("p2
vertex
knuckle"))
    P2 -- "v2 = p3 − p2" --- P3(("p3
end"))

The triplet table
#

self.finger_triplets = {
    "thumb":  [(0, 1, 2), (1, 2, 3), (2, 3, 4)],
    "index":  [(0, 5, 6), (5, 6, 7), (6, 7, 8)],
    "middle": [(0, 9, 10), (9, 10, 11), (10, 11, 12)],
    "ring":   [(0, 13, 14), (13, 14, 15), (14, 15, 16)],
    "pinky":  [(0, 17, 18), (17, 18, 19), (18, 19, 20)]
}

Landmark 0 — the wrist — starts every finger’s first triplet.

Finger	Triplet	Vertex	Joint measured
Index	(0, 5, 6)	5	MCP — joins finger to palm
Index	(5, 6, 7)	6	PIP — middle knuckle
Index	(6, 7, 8)	7	DIP — fingertip knuckle
Thumb	(0, 1, 2)	1	CMC — saddle joint at the wrist
Thumb	(1, 2, 3)	2	MCP
Thumb	(2, 3, 4)	3	IP

Why every first triplet starts at 0. The palm has no landmark of its own, so the wrist → knuckle line stands in for the metacarpal bone. It isn’t exactly collinear with a straight finger — ring and pinky metacarpals fan outward — so a relaxed straight finger rarely measures a full $\pi$. That’s part of why the calibrated “straight” threshold is 3.10 rad rather than 3.14.

From finger flexion to tendon force: linear interpolation onto a MuJoCo motor

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

On the far side of the ROS 2 topic, five flexions arrive and five tendon motors wait. One line of linear interpolation connects them — plus a name lookup that can fail silently, and a string that is allowed to push.

The formula
#

$$F(t) = F_\text{open} + t\,(F_\text{closed} - F_\text{open}) = 50 + t\,(-50 - 50) = 50 - 100\,t$$

FORCE_OPEN = 50.0
FORCE_CLOSED = -50.0

def _lerp(self, start_val, end_val, t):
    return start_val + t * (end_val - start_val)

def apply_flexions(self, flexions):
    for finger, flexion_amount in flexions.items():
        target_force = self._lerp(FORCE_OPEN, FORCE_CLOSED, flexion_amount)
        self.data.ctrl[self.motors[finger]] = target_force

Three parts of one line
#

With $t = 0.75$, a finger 75% closed:

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Twenty-one points per frame, on a laptop CPU, from a flat RGB image. That is what MediaPipe hands to a robot. It is easy to treat as a black box — until you compute angles from it and discover the box stretched your coordinate space.

Where MediaPipe sits
#

flowchart TB
    CAM["USB webcam
640×480 @ 30 fps · YUYV"] -->|"BGR frame"| RGB["cv2.cvtColor
BGR → RGB"]
    RGB --> MP["MediaPipe Hands
palm detector + landmark model"]
    MP -->|"21 × (x, y, z)"| ANG["Triplet angles
Part 5"]
    ANG --> AVG["Average per finger
Part 6"]
    AVG --> NORM["Flexion 0..1
Parts 7–8"]
    NORM -->|"/hand/target_flexions"| TWIN["MuJoCo twin"]

The vision layer knows nothing about MuJoCo. Its entire output is five numbers between 0.0 (open) and 1.0 (closed).

Two networks, not one
#

MediaPipe Hands is a cascade:

BlazePalm, a single-shot detector, finds a palm bounding box in the full frame. Palms, not hands: a palm is close to a rigid square; a hand with moving fingers is not.
A landmark model crops that region and regresses 21 keypoints, a hand-presence score and handedness.

In video mode (static_image_mode=False) the detector barely runs. Landmarks from frame t define the crop for frame t+1, and the detector wakes only when tracking confidence drops. That shortcut is why this pipeline holds 30 fps on a CPU.

Normalizing finger curl: from radians to a 0–1 flexion

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Radians belong to the camera. Newtons belong to the robot. The number that crosses between them is a plain percentage — and producing it takes one inverted formula and one clip.

The formula
#

RAW_STRAIGHT_ANGLE = 3.10  # ~177.6° — open finger
RAW_CURLED_ANGLE   = 1.60  # ~91.7° — fully curled finger

flexion = (straight_limit - avg_angle) / (straight_limit - curled_limit)
flexions[finger_name] = float(np.clip(flexion, 0.0, 1.0))

$$\text{flexion} = \operatorname{clip}\!\left(\frac{\theta_\text{straight} - \bar\theta}{\theta_\text{straight} - \theta_\text{curled}},\; 0,\; 1\right)$$

The three pieces
#

Denominator — the range. $3.10 - 1.60 = 1.50$ rad of travel. It sets the scale.

The ROS 2 topic contract: one JointState topic, two nodes, and rclpy inside a render loop

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

The whole ROS 2 layer is one topic carrying five numbers. The interesting decisions are what those numbers mean, which message type carries them, and how to run a ROS subscriber when a 3D viewer owns your main thread.

Why ROS 2 between vision and physics at all?
#

The single-process script works well. Splitting it across ROS 2 buys:

Benefit	Concretely
Process isolation	a MediaPipe crash doesn’t kill the simulator, and vice versa
Independent environments	vision and physics get their own container, dependencies and restarts
Swappable endpoints	replace the twin with a servo driver, or the tracker with a data glove
Free observability	`ros2 topic echo`, `hz`, `bag record` on the live stream
Network transparency	any machine on the LAN can subscribe

The costs — a bigger stack, DDS configuration, one more hop — are small next to 19–85 ms of inference (Part 19).

The node graph
#

flowchart LR
    subgraph C1["🐳 vision_tracker"]
        V["/vision_tracker_node
timer · 30 Hz"]
    end
    subgraph C2["🐳 mujoco_twin"]
        T["/mujoco_twin_node
spin_once in viewer loop"]
    end
    V -- "/hand/target_flexions
sensor_msgs/JointState · depth 10" --> T
    V -. "any LAN subscriber
ROS_DOMAIN_ID=42" .-> X["ros2 topic echo · rosbag
future servo driver"]

Every frame of the twin is driven by one JointState message like the one below.

The contract
#

Field	Value
Topic	`/hand/target_flexions`
Type	`sensor_msgs/msg/JointState`
Publisher	`vision_tracker_node`, timer at 30 Hz — effective rate bounded by inference
Subscriber	`mujoco_twin_node`
QoS	default reliable, keep-last 10
`header.stamp`	publisher clock at publish time
`name`	`["thumb", "index", "middle", "ring", "pinky"]`
`position`	flexion per finger, 0.0 open … 1.0 closed, same order as `name`
`velocity`, `effort`	empty

A real message, captured with ros2 topic echo during testing (published by hand with ros2 topic pub, hence the zero stamp):

Python on Mulham Fetna

Finger joint angles from three hand landmarks: the dot-product geometry

Three points make an angle #

The triplet table #

From finger flexion to tendon force: linear interpolation onto a MuJoCo motor

The formula #

Three parts of one line #

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

Where MediaPipe sits #

Two networks, not one #

Normalizing finger curl: from radians to a 0–1 flexion

The formula #

The three pieces #

The ROS 2 topic contract: one JointState topic, two nodes, and rclpy inside a render loop

Why ROS 2 between vision and physics at all? #

The node graph #

The contract #

Three points make an angle
#

The triplet table
#

The formula
#

Three parts of one line
#

Where MediaPipe sits
#

Two networks, not one
#

The formula
#

The three pieces
#

Why ROS 2 between vision and physics at all?
#

The node graph
#

The contract
#