From tutorial project to production robot: a roadmap for the tendon hand twin

Where it stands

#

1. Stage 1 · Correctness

   Measure the right thing, command the right quantity

   World landmarks. Image-normalized coordinates bend angles by up to 16° with hand orientation (Part 4). Read multi_hand_world_landmarks instead, then recalibrate.

   Tendon-length position control. Force control with zero stiffness closes a finger at −1 N (Part 11). Position actuators on tendon length gave 30° → 58° → 81° → 90° at 25/50/75/100% (kp=1000) — and it is the command a servo needs. Tracked in issue #1.

   Off the legacy MediaPipe API. mp.solutions blocks upgrades; the Tasks API HandLandmarker returns the same landmarks and world landmarks.

2. Stage 2 · Robustness

   Behave well when inputs misbehave

   Filtering — a One-Euro filter per finger: smooth when still, responsive when moving; most needed on the thumb’s narrow window. Decide what “no hand” means — today it opens the hand, which drops a held object. Hold the last value for N frames, then open. Watchdog — the twin holds its last command forever if the publisher dies; ramp to a safe pose after ~250 ms without messages. Parameters, not constants — calibration, limits, topic and rates in a mounted ROS 2 YAML file. Explicit QoS — for control, the newest sample matters most: keep-last 1 on both ends. Validate at startup — assert every mj_name2id ≥ 0 and that ctrlrange matches the code.
3. Stage 3 · Maintainability

   One implementation, tested without a camera

   One package. HandTracker and DigitalTwin live in three files, and the calibration has already drifted once. Extract a small package mounted into both containers and imported by the standalone script. Tests with no camera or display — angle maths and NaN guards, normalization and lerp, model loads with 10 tendons and 5 actuators, a steady-state sweep as a regression test, and a rosbag replayed into a headless twin (MUJOCO_GL=egl). CI — run them in the same ros:jazzy images on every push, and regenerate the doc figures when the model changes.
4. Stage 4 · Hardware

   A third node on the same topic

   Measure each finger’s real tendon stroke (the simulated index needs 42 mm), map flexion → horn angle with hard per-finger limits, rate-limit the command, and bridge through a PCA9685 PWM board or a micro-ROS microcontroller. Then publish feedback — servo current or a potentiometer — so the twin can mirror the real hand, not just the command.
5. Stage 5 · Deployment

   Least privilege and authenticated topics

   Drop privileged, run non-root with scoped X11 auth or headless; narrow ipc/pid sharing; choose LOCALHOST or SROS 2 so nobody on the LAN can move real hardware; pin base-image digests; and resolve the viewer/MediaPipe GPU contention (Part 19).

The hardware step, drawn

#

The topic contract was designed for exactly this: vision and simulation don’t change at all.

flowchart LR
    V["🐳 vision_tracker"] -- "/hand/target_flexions" --> T["🐳 mujoco_twin
simulation"]
    V -- "/hand/target_flexions" --> H["🆕 servo_driver
flexion → horn angle → PWM"]
    H --> S["5 × SG90 servos
PCA9685 or micro-ROS"]
    S -. "current · position feedback" .-> T

The design already has the servos…	…and the twin already has the fist

The one-line summary

#

Correctness first (measure in metres, command in millimetres of string), then robustness, then one tested codebase — and only then real servos, behind a watchdog and an authenticated topic.