Python

The whole ROS 2 layer is one topic carrying five numbers. The interesting decisions are what those numbers mean, which message type carries them, and how to run a ROS subscriber when a 3D viewer owns your main thread. Why ROS 2 between vision and physics at all? # The single-process script works well. Splitting it across ROS 2 buys: Benefit Concretely Process isolation a MediaPipe crash doesn’t kill the simulator, and vice versa Independent environments vision and physics get their own container, dependencies and restarts Swappable endpoints replace the twin with a servo driver, or the tracker with a data glove Free observability ros2 topic echo, hz, bag record on the live stream Network transparency any machine on the LAN can subscribe The costs — a bigger stack, DDS configuration, one more hop — are small next to 19–85 ms of inference (Part 19). The node graph # flowchart LR subgraph C1["🐳 vision_tracker"] V["/vision_tracker_node timer · 30 Hz"] end subgraph C2["🐳 mujoco_twin"] T["/mujoco_twin_node spin_once in viewer loop"] end V -- "/hand/target_flexions sensor_msgs/JointState · depth 10" --> T V -. "any LAN subscriber ROS_DOMAIN_ID=42" .-> X["ros2 topic echo · rosbag future servo driver"] Every frame of the twin is driven by one JointState message like the one below. The contract # Field Value Topic /hand/target_flexions Type sensor_msgs/msg/JointState Publisher vision_tracker_node, timer at 30 Hz — effective rate bounded by inference Subscriber mujoco_twin_node QoS default reliable, keep-last 10 header.stamp publisher clock at publish time name ["thumb", "index", "middle", "ring", "pinky"] position flexion per finger, 0.0 open … 1.0 closed, same order as name velocity, effort empty A real message, captured with ros2 topic echo during testing (published by hand with ros2 topic pub, hence the zero stamp):

Radians belong to the camera. Newtons belong to the robot. The number that crosses between them is a plain percentage — and producing it takes one inverted formula and one clip. The formula # RAW_STRAIGHT_ANGLE = 3.10 # ~177.6° — open finger RAW_CURLED_ANGLE = 1.60 # ~91.7° — fully curled finger flexion = (straight_limit - avg_angle) / (straight_limit - curled_limit) flexions[finger_name] = float(np.clip(flexion, 0.0, 1.0)) $$\text{flexion} = \operatorname{clip}\!\left(\frac{\theta_\text{straight} - \bar\theta}{\theta_\text{straight} - \theta_\text{curled}},\; 0,\; 1\right)$$ The three pieces # Denominator — the range. \(3.10 - 1.60 = 1.50\) rad of travel. It sets the scale.

Twenty-one points per frame, on a laptop CPU, from a flat RGB image. That is what MediaPipe hands to a robot. It is easy to treat as a black box — until you compute angles from it and discover the box stretched your coordinate space. Where MediaPipe sits # flowchart TB CAM["USB webcam 640×480 @ 30 fps · YUYV"] -->|"BGR frame"| RGB["cv2.cvtColor BGR → RGB"] RGB --> MP["MediaPipe Hands palm detector + landmark model"] MP -->|"21 × (x, y, z)"| ANG["Triplet angles Part 5"] ANG --> AVG["Average per finger Part 6"] AVG --> NORM["Flexion 0..1 Parts 7–8"] NORM -->|"/hand/target_flexions"| TWIN["MuJoCo twin"] The vision layer knows nothing about MuJoCo. Its entire output is five numbers between 0.0 (open) and 1.0 (closed). Two networks, not one # MediaPipe Hands is a cascade: BlazePalm, a single-shot detector, finds a palm bounding box in the full frame. Palms, not hands: a palm is close to a rigid square; a hand with moving fingers is not. A landmark model crops that region and regresses 21 keypoints, a hand-presence score and handedness. In video mode (static_image_mode=False) the detector barely runs. Landmarks from frame t define the crop for frame t+1, and the detector wakes only when tracking confidence drops. That shortcut is why this pipeline holds 30 fps on a CPU.

On the far side of the ROS 2 topic, five flexions arrive and five tendon motors wait. One line of linear interpolation connects them — plus a name lookup that can fail silently, and a string that is allowed to push. The formula # $$F(t) = F_\text{open} + t\,(F_\text{closed} - F_\text{open}) = 50 + t\,(-50 - 50) = 50 - 100\,t$$FORCE_OPEN = 50.0 FORCE_CLOSED = -50.0 def _lerp(self, start_val, end_val, t): return start_val + t * (end_val - start_val) def apply_flexions(self, flexions): for finger, flexion_amount in flexions.items(): target_force = self._lerp(FORCE_OPEN, FORCE_CLOSED, flexion_amount) self.data.ctrl[self.motors[finger]] = target_force Three parts of one line # With \(t = 0.75\), a finger 75% closed:

Every knuckle angle in this project comes from three landmarks and one dot product. No learning, no lookup table — just the definition of the angle between two vectors, plus two guards that keep a single glitchy frame from sending NaN into a motor command. Three points make an angle # An angle needs a vertex and two rays. A knuckle is the vertex; the two bones meeting there are the rays: a base point, where the previous bone starts, the vertex — the knuckle being measured, an end point, where the next bone ends. flowchart LR P1(("p1 base")) -- "v1 = p1 − p2" --- P2(("p2 vertex knuckle")) P2 -- "v2 = p3 − p2" --- P3(("p3 end")) The triplet table # self.finger_triplets = { "thumb": [(0, 1, 2), (1, 2, 3), (2, 3, 4)], "index": [(0, 5, 6), (5, 6, 7), (6, 7, 8)], "middle": [(0, 9, 10), (9, 10, 11), (10, 11, 12)], "ring": [(0, 13, 14), (13, 14, 15), (14, 15, 16)], "pinky": [(0, 17, 18), (17, 18, 19), (18, 19, 20)] } Landmark 0 — the wrist — starts every finger’s first triplet. Finger Triplet Vertex Joint measured Index (0, 5, 6) 5 MCP — joins finger to palm Index (5, 6, 7) 6 PIP — middle knuckle Index (6, 7, 8) 7 DIP — fingertip knuckle Thumb (0, 1, 2) 1 CMC — saddle joint at the wrist Thumb (1, 2, 3) 2 MCP Thumb (2, 3, 4) 3 IP Why every first triplet starts at 0. The palm has no landmark of its own, so the wrist → knuckle line stands in for the metacarpal bone. It isn’t exactly collinear with a straight finger — ring and pinky metacarpals fan outward — so a relaxed straight finger rarely measures a full \(\pi\). That’s part of why the calibrated “straight” threshold is 3.10 rad rather than 3.14.