Skip to main content
  1. Projects/
  2. Tendon-Driven Robotic Hand — A Vision-Teleoperated MuJoCo Digital Twin/

Finger joint angles from three hand landmarks: the dot-product geometry

Mulham Fetna
Author
Mulham Fetna
Renaissance Engineer
Table of Contents
ROS 2 Tendon-Driven Hand MuJoCo Twin - This article is part of a series.
Part 5: This Article
Every knuckle angle in this project comes from three landmarks and one dot product. No learning, no lookup table — just the definition of the angle between two vectors, plus two guards that keep a single glitchy frame from sending NaN into a motor command.

Three points make an angle
#

An angle needs a vertex and two rays. A knuckle is the vertex; the two bones meeting there are the rays:

  1. a base point, where the previous bone starts,
  2. the vertex — the knuckle being measured,
  3. an end point, where the next bone ends.
flowchart LR
    P1(("p1
base")) -- "v1 = p1 − p2" --- P2(("p2
vertex
knuckle")) P2 -- "v2 = p3 − p2" --- P3(("p3
end"))

The triplet table
#

self.finger_triplets = {
    "thumb":  [(0, 1, 2), (1, 2, 3), (2, 3, 4)],
    "index":  [(0, 5, 6), (5, 6, 7), (6, 7, 8)],
    "middle": [(0, 9, 10), (9, 10, 11), (10, 11, 12)],
    "ring":   [(0, 13, 14), (13, 14, 15), (14, 15, 16)],
    "pinky":  [(0, 17, 18), (17, 18, 19), (18, 19, 20)]
}
MediaPipe landmark indices used to build the triplets
Landmark 0 — the wrist — starts every finger’s first triplet.
Finger Triplet Vertex Joint measured
Index (0, 5, 6) 5 MCP — joins finger to palm
Index (5, 6, 7) 6 PIP — middle knuckle
Index (6, 7, 8) 7 DIP — fingertip knuckle
Thumb (0, 1, 2) 1 CMC — saddle joint at the wrist
Thumb (1, 2, 3) 2 MCP
Thumb (2, 3, 4) 3 IP

Why every first triplet starts at 0. The palm has no landmark of its own, so the wrist → knuckle line stands in for the metacarpal bone. It isn’t exactly collinear with a straight finger — ring and pinky metacarpals fan outward — so a relaxed straight finger rarely measures a full \(\pi\). That’s part of why the calibrated “straight” threshold is 3.10 rad rather than 3.14.

The function, line by line
#

def _calculate_angle(self, p1, p2, p3):
    v1 = p1 - p2                    # ray back along the previous bone
    v2 = p3 - p2                    # ray out along the next bone
    norm1 = np.linalg.norm(v1)
    norm2 = np.linalg.norm(v2)
    if norm1 < 1e-6 or norm2 < 1e-6:
        return 0.0                  # trap 1: a zero-length bone
    cosang = np.dot(v1, v2) / (norm1 * norm2)
    cosang = np.clip(cosang, -1.0, 1.0)   # trap 2: floating-point overshoot
    return float(np.arccos(cosang))

It is the definition of the dot product, rearranged:

$$\cos\theta = \frac{\vec v_1 \cdot \vec v_2}{\lVert \vec v_1 \rVert\,\lVert \vec v_2 \rVert} \qquad\Longrightarrow\qquad \theta = \arccos\!\left(\frac{\vec v_1 \cdot \vec v_2}{\lVert \vec v_1 \rVert\,\lVert \vec v_2 \rVert}\right)$$

Interior angle, not bend angle
#

The result is the interior angle between the bones, which runs the opposite way from the “bend” a physiotherapist would quote:

Finger Bones Interior \(\theta\) Bend \(\pi - \theta\)
Perfectly straight opposite directions \(\pi\) = 180°
Right angle perpendicular \(\pi/2\) = 90° 90°
Folded flat same direction 0 180°

Larger number, straighter finger. Keep that inversion in mind — it’s why the normalization in Part 7 subtracts from the straight limit.

Worked example
#

An index finger, in normalized image coordinates (z omitted for readability):

Landmark x y
5 (MCP) 0.50 0.60
6 (PIP) 0.50 0.50
7 (DIP) 0.55 0.45

For the PIP joint, triplet (5, 6, 7) with vertex 6:

$$\vec v_1 = p_5 - p_6 = (0.00,\ 0.10) \qquad \vec v_2 = p_7 - p_6 = (0.05,\ -0.05)$$$$\vec v_1 \cdot \vec v_2 = -0.005 \qquad \lVert\vec v_1\rVert = 0.100 \qquad \lVert\vec v_2\rVert = 0.0707$$$$\cos\theta = \frac{-0.005}{0.00707} = -0.707 \quad\Rightarrow\quad \theta = 2.356\ \text{rad} = 135^\circ$$

An interior angle of 135° — a 45° bend.

These are image-normalized coordinates, so the true bend differs when the finger isn’t aligned with the image axes — up to 16° (Part 4). MediaPipe’s metric world landmarks remove that distortion.

The two numerical traps
#

1. Zero-length vectors. If two landmarks coincide — a glitchy frame, or a finger foreshortened straight at the lens — a norm is zero and the division yields NaN. The guard returns 0.0. Note what that means here: 0 rad is “folded flat”, so a degenerate frame briefly reads as a curled joint. After averaging and clipping, the cost is one frame of partial flexion instead of a crash.

2. arccos outside its domain. Rounding can produce a cosine of 1.0000000002, and np.arccos of that is NaN. np.clip(cosang, -1, 1) makes it impossible. Without the clip, a perfectly straight finger intermittently produces NaN, which would propagate straight into the tendon force.

Both guards are unglamorous and non-optional in anything that ends at an actuator.

Next #

One finger now has three angles — but it only has one tendon. Part 6 collapses them.

Mulham Fetna
Author
Mulham Fetna
Renaissance Engineer
ROS 2 Tendon-Driven Hand MuJoCo Twin - This article is part of a series.
Part 5: This Article

Related

Normalizing finger curl: from radians to a 0–1 flexion

Radians belong to the camera. Newtons belong to the robot. The number that crosses between them is a plain percentage — and producing it takes one inverted formula and one clip. The formula # RAW_STRAIGHT_ANGLE = 3.10 # ~177.6° — open finger RAW_CURLED_ANGLE = 1.60 # ~91.7° — fully curled finger flexion = (straight_limit - avg_angle) / (straight_limit - curled_limit) flexions[finger_name] = float(np.clip(flexion, 0.0, 1.0)) $$\text{flexion} = \operatorname{clip}\!\left(\frac{\theta_\text{straight} - \bar\theta}{\theta_\text{straight} - \theta_\text{curled}},\; 0,\; 1\right)$$ The three pieces # Denominator — the range. \(3.10 - 1.60 = 1.50\) rad of travel. It sets the scale.

From finger flexion to tendon force: linear interpolation onto a MuJoCo motor

On the far side of the ROS 2 topic, five flexions arrive and five tendon motors wait. One line of linear interpolation connects them — plus a name lookup that can fail silently, and a string that is allowed to push. The formula # $$F(t) = F_\text{open} + t\,(F_\text{closed} - F_\text{open}) = 50 + t\,(-50 - 50) = 50 - 100\,t$$FORCE_OPEN = 50.0 FORCE_CLOSED = -50.0 def _lerp(self, start_val, end_val, t): return start_val + t * (end_val - start_val) def apply_flexions(self, flexions): for finger, flexion_amount in flexions.items(): target_force = self._lerp(FORCE_OPEN, FORCE_CLOSED, flexion_amount) self.data.ctrl[self.motors[finger]] = target_force Three parts of one line # With \(t = 0.75\), a finger 75% closed:

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

Twenty-one points per frame, on a laptop CPU, from a flat RGB image. That is what MediaPipe hands to a robot. It is easy to treat as a black box — until you compute angles from it and discover the box stretched your coordinate space. Where MediaPipe sits # flowchart TB CAM["USB webcam 640×480 @ 30 fps · YUYV"] -->|"BGR frame"| RGB["cv2.cvtColor BGR → RGB"] RGB --> MP["MediaPipe Hands palm detector + landmark model"] MP -->|"21 × (x, y, z)"| ANG["Triplet angles Part 5"] ANG --> AVG["Average per finger Part 6"] AVG --> NORM["Flexion 0..1 Parts 7–8"] NORM -->|"/hand/target_flexions"| TWIN["MuJoCo twin"] The vision layer knows nothing about MuJoCo. Its entire output is five numbers between 0.0 (open) and 1.0 (closed). Two networks, not one # MediaPipe Hands is a cascade: BlazePalm, a single-shot detector, finds a palm bounding box in the full frame. Palms, not hands: a palm is close to a rigid square; a hand with moving fingers is not. A landmark model crops that region and regresses 21 keypoints, a hand-presence score and handedness. In video mode (static_image_mode=False) the detector barely runs. Landmarks from frame t define the crop for frame t+1, and the detector wakes only when tracking confidence drops. That shortcut is why this pipeline holds 30 fps on a CPU.

Calibrating hand tracking to your own hand — with real data from a live session

The shipped thresholds were measured on one hand with one webcam. Here is how to measure yours — and what a live recording revealed about how far off “good enough” can be while the twin still looks perfect. Symptoms that call for calibration # What you see in the published flexions Cause Change A fist, but flexion stays below 1.0 your fist angle is above the curled limit raise *_CURLED_ANGLE to your fist reading Flexion hits 1.0 with the hand half closed curled limit too high lower *_CURLED_ANGLE Flexion above 0 with a relaxed open hand your open angle is below the straight limit lower *_STRAIGHT_ANGLE to your open reading The thumb flickers thumb window too narrow for your jitter widen it (Part 8) Calibrate against the numbers, not the render. With the current force-control tuning, the simulated finger snaps shut past flexion ≈ 0.51 (Part 11), so most calibration errors are invisible in the viewer. Watch ros2 topic echo /hand/target_flexions. Real data: what a live session produced # MuJoCo’s viewer has a Control panel showing the live force of every motor. Since the force is \(F = 50 - 100 \cdot \text{flexion}\), every recorded frame gives back the exact flexion the tracker published: flexion = (50 − F) / 100.

How a webcam moves a simulated tendon-driven hand

One vision container turns webcam frames into five numbers. One simulation container turns those numbers into tendon forces. Everything else in this series is detail inside one of those two boxes — or the pipe between them. Left to right, three layers in one frame: vision (landmarks), simulation (the twin), actuation (live motor forces from MuJoCo’s Control panel). End to end # flowchart LR subgraph VISION["🐳 vision_tracker container"] direction TB A["Webcam frame 640×480 BGR"] --> B["MediaPipe Hands 21 landmarks"] B --> C["3 knuckle angles / finger dot product"] C --> D["mean → 1 curl angle (underactuation)"] D --> E["normalize + clip flexion 0..1"] end subgraph TWIN["🐳 mujoco_twin container"] direction TB F["lerp +50 N … −50 N"] --> G["data.ctrl on pull_{finger} motor"] G --> H["spatial tendon through 6 sites"] H --> I["3 passive hinge joints curl"] I --> J["MuJoCo viewer"] end E -- "ROS 2 · /hand/target_flexions sensor_msgs/JointState" --> F Stage What comes out Deep dive MediaPipe Hands 21 (x, y, z) landmarks per frame Part 4 Triplet angles 3 interior angles per finger, in radians Part 5 Averaging 1 curl angle per finger Part 6 Normalization flexion 0..1 (the thumb has its own window) Parts 7–8 ROS 2 topic JointState: names are fingers, positions are flexions Part 15 Lerp force in newtons per tendon Part 10 Tendon physics joint angles Part 11 Why the pipe carries flexions, not angles or forces # The contract between the containers is five unitless numbers: 0.0 is an open finger, 1.0 is a closed one. That choice is the architecture.