Mujoco

Squeeze your thumb across your palm as hard as you can and, by the finger thresholds, it is only half closed. The thumb isn’t a finger with a shorter bone — it’s a different joint. Thumb curled, fingers open. Only pull_thumb goes negative (−19.5 N, flexion 0.69); every other motor pushes open. Hinges versus a saddle # Index to pinky are chains of hinge joints. A fist rolls them into a tight spiral, each knuckle approaching 90°. The thumb hangs from the carpometacarpal (CMC) saddle joint at the wrist. It sweeps across the palm — opposition — instead of simply curling. Much of “closing the thumb” is rotation of the whole thumb, not bending at its knuckles. What MediaPipe measures because of it # Interior angles come from 3D landmark positions, so the sweep only partly shows up as knuckle bend: Open Fully closed Travel Finger — mean of MCP/PIP/DIP ≈ 3.10 rad ≈ 1.60 rad 1.50 rad Thumb — mean of CMC/MCP/IP ≈ 2.90 rad ≈ 2.30 rad 0.60 rad A fully closed thumb stops near 2.30 rad (~132°). Even its open angle sits below a finger’s — a relaxed thumb is never in line with the wrist.

The vision layer sends a carefully normalized, smooth 0-to-1 signal. The simulated finger turns it into two states: open and closed. This is the most important thing to understand about the twin — and every number below was measured on the model in the repository. The measurement # The same flexion was applied to all five motors, the model simulated for 3 s from rest, and the steady-state joint angles recorded — 41 flexion values from 0 to 1. Flexion Force Index MCP / PIP / DIP Thumb CMC / MP / IP 0.000 +50.0 N −3.3° / −3.2° / −3.3° −2.9° / −2.7° / −2.7° 0.250 +25.0 N −1.7° / −1.6° / −1.7° −1.5° / −1.3° / −1.4° 0.500 0.0 N 0.0° / 0.0° / 0.0° 0.0° / 0.0° / 0.0° 0.625 −12.5 N 91.1° / 91.0° / 91.0° 90.6° / 53.9° / 45.6° 1.000 −50.0 N 94.0° / 93.8° / 93.9° 92.3° / 54.2° / 46.0° Joint limits are 90°; values just above it are MuJoCo’s soft limit being pressed.

The single-script version felt smoother than the containers. The obvious suspect was Docker. The measurements say otherwise — and point at something more interesting: two programs sharing one integrated GPU. Setup # Laptop with an Intel Comet Lake CPU (12 threads) and Intel UHD Graphics; Linux on Wayland with XWayland; USB webcam at 640×480, 30 fps, YUYV. Each row: mean of 150 frames after a 10-frame warm-up, no hand in view. Measured 2026-09-16. The result # Condition Camera read MediaPipe imshow Loop Vision alone, container 10.0 ms 19.0 ms 4.5 ms 29.8 Hz Vision alone, host venv (MediaPipe 0.10.11) 10.6 ms 19.2 ms 3.9 ms 29.7 Hz Container vision + 1-core CPU burner 8.2 ms 23.5 ms 3.9 ms 28.1 Hz Host vision + plain MuJoCo viewer on host (no Docker, no ROS) 1.6 ms 54.9 ms 5.9 ms 16.0 Hz Container vision + mujoco_twin capped at 30 Hz 1.7 ms 63.7 ms 8.1 ms 13.6 Hz Container vision + mujoco_twin as shipped 1.9 ms 85.6 ms 10.2 ms 10.2 Hz End to end, ros2 topic hz /hand/target_flexions read 30.0 Hz with only vision_tracker running and 10.7 Hz once mujoco_twin started.

avg_angle = np.mean(angles) looks like noise filtering. It isn’t. It is a mechanical decision: a human finger has three joints you can move separately, and this robot finger has one string. Not smoothing — compression # “Averaging” in a sensor pipeline usually means averaging over time to reduce noise. Nothing here keeps history between frames. The mean is taken across space — over the three joints of one finger in a single frame — to solve a problem called underactuation. The 3-to-1 problem # Human finger Robot finger Joints 3 (MCP, PIP, DIP) 3 hinges (*_mcp, *_pip, *_dip) Independent actuators many muscles; joints move semi-independently 1 flexor tendon, 1 motor Degrees of freedom you can command ~3 1 A system with fewer actuators than joints is underactuated. A single tendon threads all three joints of each robot finger, so the only command is “pull this string with force F” — and the joints share that pull according to routing geometry and dynamics. One string, six via-points, three joints. Pull it and all three knuckles move together. So the vision layer must compress three human measurements into one robot command.

Every entry here was hit, or deliberately checked, while building this project. Error messages are quoted exactly so a search for the message lands on the fix. Camera and windows # RuntimeError: Failed to open camera at index 0 # Something else holds the webcam — the standalone script, a previous container, a browser tab. Run docker compose down and close video apps. A camera opens in one process at a time. The camera is another node. ls /dev/video*, then map that device and set CAMERA_INDEX (many webcams expose /dev/video0 for frames and /dev/video1 for metadata — use the first). The device isn’t mapped. Check with docker compose config | grep video. Windows don’t open — cannot connect to X server, could not connect to display # Run ./setup_host.sh (it runs xhost +local:root). The permission resets when you log out. Make sure echo $DISPLAY on the host prints :0, or export DISPLAY before docker compose up. On Wayland, confirm XWayland is running: ls /tmp/.X11-unix/ should list X0. Black, blank or garbled OpenCV window # QT_X11_NO_MITSHM=1 must reach the container. It’s in the shared Compose environment — if a service defines its own environment:, it must merge the shared anchor with <<: *ros-env rather than replace it (Part 16).

The whole ROS 2 layer is one topic carrying five numbers. The interesting decisions are what those numbers mean, which message type carries them, and how to run a ROS subscriber when a 3D viewer owns your main thread. Why ROS 2 between vision and physics at all? # The single-process script works well. Splitting it across ROS 2 buys: Benefit Concretely Process isolation a MediaPipe crash doesn’t kill the simulator, and vice versa Independent environments vision and physics get their own container, dependencies and restarts Swappable endpoints replace the twin with a servo driver, or the tracker with a data glove Free observability ros2 topic echo, hz, bag record on the live stream Network transparency any machine on the LAN can subscribe The costs — a bigger stack, DDS configuration, one more hop — are small next to 19–85 ms of inference (Part 19). The node graph # flowchart LR subgraph C1["🐳 vision_tracker"] V["/vision_tracker_node timer · 30 Hz"] end subgraph C2["🐳 mujoco_twin"] T["/mujoco_twin_node spin_once in viewer loop"] end V -- "/hand/target_flexions sensor_msgs/JointState · depth 10" --> T V -. "any LAN subscriber ROS_DOMAIN_ID=42" .-> X["ros2 topic echo · rosbag future servo driver"] Every frame of the twin is driven by one JointState message like the one below. The contract # Field Value Topic /hand/target_flexions Type sensor_msgs/msg/JointState Publisher vision_tracker_node, timer at 30 Hz — effective rate bounded by inference Subscriber mujoco_twin_node QoS default reliable, keep-last 10 header.stamp publisher clock at publish time name ["thumb", "index", "middle", "ring", "pinky"] position flexion per finger, 0.0 open … 1.0 closed, same order as name velocity, effort empty A real message, captured with ros2 topic echo during testing (published by hand with ros2 topic pub, hence the zero stamp):

A raw CAD export gives you bodies, joints, meshes and sites — and a hand that does nothing. Four edits and one extra file turn it into a tendon-driven twin. One of those edits, it turns out, does nothing at all — which is worth understanding too. Which edits survive a re-export # flowchart TB subgraph generated["robot.xml — regenerated by onshape-to-robot"] D["① joint defaults manual — re-apply"] T["② tendons + ③ contacts auto-injected from tendons.xml"] B["bodies · joints · sites · meshes generated"] A["④ actuators manual — re-apply"] end S["⑤ scene.xml floor · lights · skybox never regenerated"] -->|"include robot.xml"| generated # Edit Lives in Survives re-export? ① Joint friction / armature / damping defaults robot.xml <default> ✘ re-apply ② Flexor + extensor spatial tendons tendons.xml → injected ✔ ③ Contact exclusions tendons.xml → injected ✔ ④ Five tendon motors robot.xml <actuator> ✘ re-apply ⑤ Environment scene.xml ✔ separate file tendons.xml is kept byte-identical to the <tendon> and <contact> region of robot.xml. Tune a tendon in robot.xml without copying it back and the next export silently reverts it. ① Joint defaults — stability # <default class="ros2-tendon-driven-hand-gazebo-digital-twin"> <joint frictionloss="0.001" armature="0.0001" damping="0.01"/> Attribute Value Role damping 0.01 N·m·s/rad stops a 2 g phalanx reaching absurd speed when 50 N yanks it armature 0.0001 kg·m² rotor-like inertia on each joint’s diagonal — conditions the solver for very light bodies frictionloss 0.001 N·m a small dry-friction dead-band so joints settle instead of creeping Phalanges weigh 1.8–5.4 g. Without these, tiny inertias under large tendon forces blow up the integrator — an earlier, larger revision of the model logged Nan, Inf or huge value in QACC at DOF 128. The simulation is unstable.

Clone, run one setup script, run one Compose command — and a webcam window and a MuJoCo viewer open side by side, with a simulated hand that closes when you close yours. Here are four ways in, from the full stack down to the bare model. The goal of this article: both windows open, and the twin follows your hand. What you need # Requirement Check with Notes Linux desktop, X11 or XWayland echo $DISPLAY → :0 Wayland sessions work through XWayland Docker Engine + Compose v2 docker compose version Built with Docker 29.8 / Compose 5.5 A webcam ls /dev/video0 Close any other app using it GPU device nodes ls /dev/dri Intel/AMD out of the box; NVIDIA needs the container toolkit (Part 18) ~5 GB of disk images: vision 2.9 GB, twin 1.7 GB No ROS installation is needed on the host — ROS 2 Jazzy lives inside the containers. Four ways to run it # Docker Compose (full stack) Single Python script Twin only, no camera Model viewer git clone https://github.com/mulhamfetna/ros2-tendon-driven-hand-mujoco-digital-twin-vision-teleoperation.git cd ros2-tendon-driven-hand-mujoco-digital-twin-vision-teleoperation ./setup_host.sh # X11 access for the containers + camera/GPU checks (once per login) docker compose up --build # builds both images, starts vision_tracker and mujoco_twin Stop with Ctrl+C, then docker compose down to release the camera.

A fixed palm, five three-segment digits, five free-spinning servo horns, ten strings through seventy points, five motors. Every number here was read from the compiled model with MuJoCo 3.13.0. Open (+50 N) Neutral (0 N) Fist (−50 N) Index tendon only Middle tendon only Ring tendon only Pinky tendon only Thumb tendon only Previous Next At a glance # Quantity Value Notes Bodies 22 world, palm, 15 phalanges, 5 servo horns Joints / DoF 20 / 20 all hinges; no free joint — the palm is welded to the world Knuckle joints 15 passive, 90° limits Servo horn joints 5 unlimited and unactuated Geoms 87 43 visual + 43 collision meshes + floor Meshes 14 STL Sites 70 tendon via-points and anchors Tendons 10 5 flexor (actuated) + 5 extensor (passive) Actuators 5 <motor> on flexors, ±50 N Timestep / integrator 0.002 s / Euler 500 steps per simulated second Total mass 0.398 kg palm block 341.8 g The kinematic tree # flowchart TB W["world"] --> P["part_1 · palm + servo block 341.8 g · 20 sites"] P --> I1["part_2_4 · index proximal 4.6 g · index_mcp"] --> I2["part_3_4 1.8 g · index_pip"] --> I3["part_4_4 4.6 g · index_dip"] P --> M1["part_2_3 · middle middle_mcp"] --> M2["part_3_3 middle_pip"] --> M3["part_4_3 middle_dip"] P --> R1["part_2_2 · ring ring_mcp"] --> R2["part_3_2 ring_pip"] --> R3["part_4_2 ring_dip"] P --> K1["part_2 · pinky pinky_mcp"] --> K2["part_3 pinky_pip"] --> K3["part_4 pinky_dip"] P --> T1["part_5 · thumb 5.4 g · thumb_cmc"] --> T2["part_6 2.4 g · thumb_mp"] --> T3["part_7 2.5 g · thumb_ip"] P --> H["servo_horn … servo_horn_5 0.3 g each"] Body names are CAD part names, not finger names. Repeated instances get _2, _3, … appended, so the pinky chain is part_2 → part_3 → part_4 and the index chain is part_2_4 → part_3_4 → part_4_4. Address joints, sites, tendons and actuators in code — never bodies. Joints and their signs # Joint Range (rad) Bend direction index_mcp, index_pip, index_dip [−1.571, 0] negative middle_mcp [−1.571, 0] negative middle_pip, middle_dip [0, +1.571] positive ring_mcp, ring_dip [−1.571, 0] negative ring_pip [0, +1.571] positive pinky_mcp, pinky_dip [−1.571, 0] negative pinky_pip [0, +1.571] positive thumb_cmc, thumb_mp, thumb_ip [−1.571, 0] negative servo_* × 5 unlimited — Bend direction is inconsistent, because each Onshape mate’s axis was exported as-is. Anything that reads joint angles must normalize per joint:

One vision container turns webcam frames into five numbers. One simulation container turns those numbers into tendon forces. Everything else in this series is detail inside one of those two boxes — or the pipe between them. Left to right, three layers in one frame: vision (landmarks), simulation (the twin), actuation (live motor forces from MuJoCo’s Control panel). End to end # flowchart LR subgraph VISION["🐳 vision_tracker container"] direction TB A["Webcam frame 640×480 BGR"] --> B["MediaPipe Hands 21 landmarks"] B --> C["3 knuckle angles / finger dot product"] C --> D["mean → 1 curl angle (underactuation)"] D --> E["normalize + clip flexion 0..1"] end subgraph TWIN["🐳 mujoco_twin container"] direction TB F["lerp +50 N … −50 N"] --> G["data.ctrl on pull_{finger} motor"] G --> H["spatial tendon through 6 sites"] H --> I["3 passive hinge joints curl"] I --> J["MuJoCo viewer"] end E -- "ROS 2 · /hand/target_flexions sensor_msgs/JointState" --> F Stage What comes out Deep dive MediaPipe Hands 21 (x, y, z) landmarks per frame Part 4 Triplet angles 3 interior angles per finger, in radians Part 5 Averaging 1 curl angle per finger Part 6 Normalization flexion 0..1 (the thumb has its own window) Parts 7–8 ROS 2 topic JointState: names are fingers, positions are flexions Part 15 Lerp force in newtons per tendon Part 10 Tendon physics joint angles Part 11 Why the pipe carries flexions, not angles or forces # The contract between the containers is five unitless numbers: 0.0 is an open finger, 1.0 is a closed one. That choice is the architecture.

Containers are headless by design. This project needs two windows, a webcam and a GPU — so the Compose file spends most of its lines punching carefully chosen holes back through the isolation. Two containers, two windows, one X display: the Qt/OpenCV tracker (vision_tracker) and the GLFW MuJoCo viewer (mujoco_twin), recorded on Wayland through XWayland. Windows: pass the X11 socket through # environment: DISPLAY: ${DISPLAY:-:0} QT_X11_NO_MITSHM: 1 volumes: - /tmp/.X11-unix:/tmp/.X11-unix:rw Piece What it does /tmp/.X11-unix mount the X server listens on a Unix socket here (X0 for :0); mounting it gives the container a line to it DISPLAY tells X clients (GLFW, Qt) which display to use; defaults to :0 QT_X11_NO_MITSHM=1 stops Qt — OpenCV’s imshow backend — from using MIT-SHM, which fails across containers and gives blank or garbled windows xhost +local:root (in setup_host.sh) the X server refuses untrusted clients; this admits local root, which is who the containers run as On Wayland # Wayland sessions (KDE Plasma, GNOME) still run XWayland on :0, and both windows open through it — the recording above was made exactly that way. Running MuJoCo natively on a Wayland host, GLFW may warn Wayland: The platform does not provide the window position; it’s harmless.

On the far side of the ROS 2 topic, five flexions arrive and five tendon motors wait. One line of linear interpolation connects them — plus a name lookup that can fail silently, and a string that is allowed to push. The formula # $$F(t) = F_\text{open} + t\,(F_\text{closed} - F_\text{open}) = 50 + t\,(-50 - 50) = 50 - 100\,t$$FORCE_OPEN = 50.0 FORCE_CLOSED = -50.0 def _lerp(self, start_val, end_val, t): return start_val + t * (end_val - start_val) def apply_flexions(self, flexions): for finger, flexion_amount in flexions.items(): target_force = self._lerp(FORCE_OPEN, FORCE_CLOSED, flexion_amount) self.data.ctrl[self.motors[finger]] = target_force Three parts of one line # With \(t = 0.75\), a finger 75% closed:

The simulated hand was never modelled by hand. It is an Onshape assembly — five SG90 servos, fifteen knuckle mates, a palm full of tendon channels — pulled through the Onshape API and written out as MuJoCo XML. Here is the design, and every setting that steers the export. Open the Onshape assembly The design # Your browser cannot play this video. Download video. Palm and fingers: four three-phalanx fingers and a three-segment thumb, every knuckle a revolute mate with limits. The RGB triads in the views are mate connectors. Tendon channels: one per finger, running down the palm into the base. Servo block: five SG90-class servos, staggered so each horn sits under a tendon exit. The design has a history # Start 2026-09-02 The first version in the history. v1.0.0 — MediaPipe 2026-09-06 The joint-angle-driven hand behind the RViz predecessor project. v1.0.1 → Main 2026-09-08 Point release and the main line the later work branches from. V3 → Mujoco branch 2026-09-16 The current design used by this twin — the version with the servo base block shown above. Onshape version history Mate features 43 part instances, 112 mate features: the 15 dof_* knuckle mates, the servo mates, and many Fastened mates.