↓ Skip to main content
Mulham Fetna
  1. Projects/
  2. Tendon-Driven Robotic Hand — A Vision-Teleoperated MuJoCo Digital Twin/

Troubleshooting and FAQ: MuJoCo, MediaPipe and ROS 2 in Docker

· ·
Projects Robotics Mujoco Ros2 Docker Troubleshooting
Mulham Fetna
Author
Mulham Fetna
Renaissance Engineer
Table of Contents
ROS 2 Tendon-Driven Hand MuJoCo Twin - This article is part of a series.
Part 1: Run the tendon-driven hand digital twin in 10 minutes
Part 2: How a webcam moves a simulated tendon-driven hand
Part 3: This Article
Part 4: MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks
Part 5: Finger joint angles from three hand landmarks: the dot-product geometry
Part 6: Underactuation: why three knuckle angles become one tendon command
Part 7: Normalizing finger curl: from radians to a 0–1 flexion
Part 8: Why the thumb needs its own thresholds in hand tracking
Part 9: Calibrating hand tracking to your own hand — with real data from a live session
Part 10: From finger flexion to tendon force: linear interpolation onto a MuJoCo motor
Part 11: Why the simulated tendon finger is a switch — and three fixes tested in MuJoCo
Part 12: From an Onshape assembly to a MuJoCo model with onshape-to-robot
Part 13: The manual MJCF edits that turn a CAD export into a tendon-driven hand
Part 14: Inside the MuJoCo tendon hand model: bodies, joints, sites and sign conventions
Part 15: The ROS 2 topic contract: one JointState topic, two nodes, and rclpy inside a render loop
Part 16: Docker Compose architecture for ROS 2: dependency-only images and mounted code
Part 17: ROS 2 across Docker containers: shared network, discovery and Fast DDS shared memory
Part 18: GUI, webcam and GPU passthrough for ROS 2 and MuJoCo containers
Part 19: What limits the frame rate: MediaPipe, Docker and a MuJoCo viewer on one laptop
Part 20: From tutorial project to production robot: a roadmap for the tendon hand twin
Every entry here was hit, or deliberately checked, while building this project. Error messages are quoted exactly so a search for the message lands on the fix.

Camera and windows
#

RuntimeError: Failed to open camera at index 0
#

  • Something else holds the webcam — the standalone script, a previous container, a browser tab. Run docker compose down and close video apps. A camera opens in one process at a time.
  • The camera is another node. ls /dev/video*, then map that device and set CAMERA_INDEX (many webcams expose /dev/video0 for frames and /dev/video1 for metadata — use the first).
  • The device isn’t mapped. Check with docker compose config | grep video.

Windows don’t open — cannot connect to X server, could not connect to display
#

  • Run ./setup_host.sh (it runs xhost +local:root). The permission resets when you log out.
  • Make sure echo $DISPLAY on the host prints :0, or export DISPLAY before docker compose up.
  • On Wayland, confirm XWayland is running: ls /tmp/.X11-unix/ should list X0.

Black, blank or garbled OpenCV window
#

QT_X11_NO_MITSHM=1 must reach the container. It’s in the shared Compose environment — if a service defines its own environment:, it must merge the shared anchor with <<: *ros-env rather than replace it (Part 16).

The MuJoCo viewer crawls
#

The GPU isn’t reaching the container and Mesa fell back to software rendering. Check /dev/dri is mapped; on NVIDIA use the NVIDIA Container Toolkit (Part 18).

ROS 2 communication
#

Landmarks are drawn, but the twin doesn’t move
# 

docker compose exec mujoco_twin bash -c 'source /opt/ros/jazzy/setup.bash && ros2 topic hz /hand/target_flexions'
  • No rate is printed — discovery failed. Both services need the same ROS_DOMAIN_ID and network_mode: host.
  • A rate is printed — messages arrive; inspect values with ros2 topic echo. Flexions under 0.5 keep a finger open by design (Part 11); if your fist doesn’t reach ~0.6, recalibrate (Part 9).

Another computer can’t see /hand/target_flexions
#

Same ROS_DOMAIN_ID=42, same subnet, host firewall open for UDP 17900–17930, and no Wi-Fi client isolation or multicast filtering. Details in Part 17.

Python environment
#

ModuleNotFoundError: No module named 'mujoco' (or mediapipe) in a working venv
#

A sourced ROS installation prepends its own Python paths through PYTHONPATH, shadowing the venv. Run with env -u PYTHONPATH venv/bin/python ….

AttributeError: module 'mediapipe' has no attribute 'solutions'
#

You installed a MediaPipe release without the legacy API this project uses. Pin mediapipe==0.10.14 (container, Python 3.12) or 0.10.11 (standalone, Python 3.10).

MuJoCo model
#

Nan, Inf or huge value in QACC at DOF … The simulation is unstable.
#

Almost always a re-exported robot.xml that lost the joint defaults — damping, armature, frictionloss. Phalanges here weigh 1.8–5.4 g; without them, a 50 N tendon makes the integrator explode. Re-apply them (Part 13).

Error: site 'palm_in_flex_index' not found in wrap 1
#

A CAD reference frame was renamed or removed, and tendons.xml threads tendons through sites by name. Restore the name in Onshape or update tendons.xml.

A finger moves the wrong way — or the wrong finger moves
#

An actuator was renamed. mj_name2id returned -1, and data.ctrl[-1] wrote to the last actuator instead of failing. Check the names are pull_{thumb,index,middle,ring,pinky}.

Harmless log noise
#

Message From Action
QFontDatabase: Cannot find font directory …/cv2/qt/fonts OpenCV’s bundled Qt ignore
SymbolDatabase.GetPrototype() is deprecated protobuf inside MediaPipe 0.10.14 ignore
Created TensorFlow Lite XNNPACK delegate for CPU MediaPipe ignore
GLFWError: (65548) Wayland: The platform does not provide the window position GLFW on a native Wayland host ignore
Failed to load plugin 'libdecor-gtk.so' GLFW decorations on Wayland ignore

FAQ
#

What is a tendon-driven robotic hand?
A tendon-driven robotic hand moves its finger joints with strings (tendons) routed through the fingers and pulled by actuators in the palm or forearm, similar to human flexor tendons. In Mulham Fetna’s open-source MuJoCo digital twin, each finger has one flexor tendon running through all three knuckles, so the hand is underactuated: 15 finger joints driven by 5 actuators.
What does digital twin mean in the tendon-driven hand project?
It means a MuJoCo physics simulation of the hand, generated from the same Onshape CAD model as the physical design, that mirrors the operator’s hand pose in real time from a webcam. It currently mirrors the commands sent to the hand; mirroring measured state from real servos is the planned hardware step.
Why use MuJoCo instead of Gazebo for a tendon-driven hand?
MuJoCo has first-class spatial tendons routed through named sites, actuators that act directly on tendons, and a fast, stable solver for small and very light articulated bodies such as 2-gram finger phalanges. That is exactly what a string-driven finger needs, and MuJoCo installs with a single pip command.
Why does the simulated tendon hand only fully open or fully close?
Because the fingers are force-controlled and the finger joints have zero stiffness, so nothing balances the tendon pull: about −1 N of tendon force already closes a finger to its 90-degree stop. Switching to position control of tendon length (a MuJoCo position actuator with kp around 1000) produced a smooth, proportional curl in testing.
Is the Docker version of the project slower than running it natively?
Not because of Docker. MediaPipe hand tracking measured 19.0 ms per frame inside the container and 19.2 ms in a native Python environment. It slows to 55–86 ms whenever a MuJoCo viewer renders at the same time, with or without Docker; contention on the shared integrated GPU is the leading, unconfirmed hypothesis.
Can the project control a real robotic hand?
It is designed to: the vision container publishes normalized finger flexions on the ROS 2 topic /hand/target_flexions, so a servo-driver node can subscribe to the same topic and drive real servos without changing the vision or simulation code. The hardware node is not implemented yet.
Does the tendon hand digital twin run on Windows or macOS?
The Python logic does, through the standalone single-process script. The Docker Compose setup relies on Linux device nodes (/dev/video0, /dev/dri), the X11 socket and host networking, so on Windows or macOS use the standalone script, or WSL2 with WSLg and USB passthrough (untested).
Mulham Fetna
Author
Mulham Fetna
Renaissance Engineer
ROS 2 Tendon-Driven Hand MuJoCo Twin - This article is part of a series.
Part 1: Run the tendon-driven hand digital twin in 10 minutes
Part 2: How a webcam moves a simulated tendon-driven hand
Part 3: This Article
Part 4: MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks
Part 5: Finger joint angles from three hand landmarks: the dot-product geometry
Part 6: Underactuation: why three knuckle angles become one tendon command
Part 7: Normalizing finger curl: from radians to a 0–1 flexion
Part 8: Why the thumb needs its own thresholds in hand tracking
Part 9: Calibrating hand tracking to your own hand — with real data from a live session
Part 10: From finger flexion to tendon force: linear interpolation onto a MuJoCo motor
Part 11: Why the simulated tendon finger is a switch — and three fixes tested in MuJoCo
Part 12: From an Onshape assembly to a MuJoCo model with onshape-to-robot
Part 13: The manual MJCF edits that turn a CAD export into a tendon-driven hand
Part 14: Inside the MuJoCo tendon hand model: bodies, joints, sites and sign conventions
Part 15: The ROS 2 topic contract: one JointState topic, two nodes, and rclpy inside a render loop
Part 16: Docker Compose architecture for ROS 2: dependency-only images and mounted code
Part 17: ROS 2 across Docker containers: shared network, discovery and Fast DDS shared memory
Part 18: GUI, webcam and GPU passthrough for ROS 2 and MuJoCo containers
Part 19: What limits the frame rate: MediaPipe, Docker and a MuJoCo viewer on one laptop
Part 20: From tutorial project to production robot: a roadmap for the tendon hand twin

Related

Run the tendon-driven hand digital twin in 10 minutes

Projects Robotics Mujoco Ros2 Docker Tutorial
Clone, run one setup script, run one Compose command — and a webcam window and a MuJoCo viewer open side by side, with a simulated hand that closes when you close yours. Here are four ways in, from the full stack down to the bare model. The goal of this article: both windows open, and the twin follows your hand. What you need # Requirement Check with Notes Linux desktop, X11 or XWayland echo $DISPLAY → :0 Wayland sessions work through XWayland Docker Engine + Compose v2 docker compose version Built with Docker 29.8 / Compose 5.5 A webcam ls /dev/video0 Close any other app using it GPU device nodes ls /dev/dri Intel/AMD out of the box; NVIDIA needs the container toolkit (Part 18) ~5 GB of disk images: vision 2.9 GB, twin 1.7 GB No ROS installation is needed on the host — ROS 2 Jazzy lives inside the containers. Four ways to run it # Docker Compose (full stack) Single Python script Twin only, no camera Model viewer git clone https://github.com/mulhamfetna/ros2-tendon-driven-hand-mujoco-digital-twin-vision-teleoperation.git cd ros2-tendon-driven-hand-mujoco-digital-twin-vision-teleoperation ./setup_host.sh # X11 access for the containers + camera/GPU checks (once per login) docker compose up --build # builds both images, starts vision_tracker and mujoco_twin Stop with Ctrl+C, then docker compose down to release the camera.

Docker Compose architecture for ROS 2: dependency-only images and mounted code

Projects Docker Ros2 Robotics Devops
Both images hold dependencies and nothing else. The code and the robot model are mounted from your checkout at runtime — so an edit is a five-second restart, not a five-minute rebuild. The big picture # flowchart TB subgraph HOST["🐧 Linux host"] CAMDEV["/dev/video0"] GPU["/dev/dri · Intel iGPU"] X11["/tmp/.X11-unix XWayland :0"] SHM["/dev/shm Fast DDS segments"] NET["host network UDP multicast · domain 42"] REPO["repository checkout"] subgraph VT["🐳 vision_tracker · 2.9 GB image"] VN["vision_tracker_node.py mediapipe 0.10.14 · OpenCV"] end subgraph MT["🐳 mujoco_twin · 1.7 GB image"] MN["mujoco_twin_node.py mujoco 3.13.0 · GLFW"] end end CAMDEV --> VN GPU --> VN GPU --> MN X11 <--> VN X11 <--> MN VN <--> SHM <--> MN VN <--> NET <--> MN REPO -. "bind mount .:/workspace:ro" .-> VN REPO -. "bind mount .:/workspace:ro" .-> MN Repository layout # . ├── docker-compose.yml # both services, shared namespaces ├── setup_host.sh # xhost + device checks, once per login ├── vision_tracker/ │ ├── Dockerfile # ros:jazzy + mediapipe==0.10.14 │ ├── .dockerignore # src/ is mounted, so keep it out of the build context │ └── src/vision_tracker_node.py ├── mujoco_twin/ │ ├── Dockerfile # ros:jazzy + mujoco==3.13.0 │ ├── .dockerignore # src/ and model/ are mounted │ ├── src/mujoco_twin_node.py │ └── model/ # scene.xml → robot.xml (+ tendons.xml), assets/, config.json ├── standalone/main.py # the same pipeline, one process └── docs/ Each service folder is its own build context: editing the vision Dockerfile never invalidates the twin’s image cache, and neither build uploads the 13 MB of meshes it doesn’t need.

GUI, webcam and GPU passthrough for ROS 2 and MuJoCo containers

Projects Docker Linux Ros2 Mujoco
Containers are headless by design. This project needs two windows, a webcam and a GPU — so the Compose file spends most of its lines punching carefully chosen holes back through the isolation. Two containers, two windows, one X display: the Qt/OpenCV tracker (vision_tracker) and the GLFW MuJoCo viewer (mujoco_twin), recorded on Wayland through XWayland. Windows: pass the X11 socket through # environment: DISPLAY: ${DISPLAY:-:0} QT_X11_NO_MITSHM: 1 volumes: - /tmp/.X11-unix:/tmp/.X11-unix:rw Piece What it does /tmp/.X11-unix mount the X server listens on a Unix socket here (X0 for :0); mounting it gives the container a line to it DISPLAY tells X clients (GLFW, Qt) which display to use; defaults to :0 QT_X11_NO_MITSHM=1 stops Qt — OpenCV’s imshow backend — from using MIT-SHM, which fails across containers and gives blank or garbled windows xhost +local:root (in setup_host.sh) the X server refuses untrusted clients; this admits local root, which is who the containers run as On Wayland # Wayland sessions (KDE Plasma, GNOME) still run XWayland on :0, and both windows open through it — the recording above was made exactly that way. Running MuJoCo natively on a Wayland host, GLFW may warn Wayland: The platform does not provide the window position; it’s harmless.

How a webcam moves a simulated tendon-driven hand

Projects Robotics Mujoco Ros2 Computer-Vision Digital-Twin
One vision container turns webcam frames into five numbers. One simulation container turns those numbers into tendon forces. Everything else in this series is detail inside one of those two boxes — or the pipe between them. Left to right, three layers in one frame: vision (landmarks), simulation (the twin), actuation (live motor forces from MuJoCo’s Control panel). End to end # flowchart LR subgraph VISION["🐳 vision_tracker container"] direction TB A["Webcam frame 640×480 BGR"] --> B["MediaPipe Hands 21 landmarks"] B --> C["3 knuckle angles / finger dot product"] C --> D["mean → 1 curl angle (underactuation)"] D --> E["normalize + clip flexion 0..1"] end subgraph TWIN["🐳 mujoco_twin container"] direction TB F["lerp +50 N … −50 N"] --> G["data.ctrl on pull_{finger} motor"] G --> H["spatial tendon through 6 sites"] H --> I["3 passive hinge joints curl"] I --> J["MuJoCo viewer"] end E -- "ROS 2 · /hand/target_flexions sensor_msgs/JointState" --> F Stage What comes out Deep dive MediaPipe Hands 21 (x, y, z) landmarks per frame Part 4 Triplet angles 3 interior angles per finger, in radians Part 5 Averaging 1 curl angle per finger Part 6 Normalization flexion 0..1 (the thumb has its own window) Parts 7–8 ROS 2 topic JointState: names are fingers, positions are flexions Part 15 Lerp force in newtons per tendon Part 10 Tendon physics joint angles Part 11 Why the pipe carries flexions, not angles or forces # The contract between the containers is five unitless numbers: 0.0 is an open finger, 1.0 is a closed one. That choice is the architecture.

ROS 2 across Docker containers: shared network, discovery and Fast DDS shared memory

Projects Ros2 Docker Networking Robotics
“The topic shows up in ros2 topic list, but echo prints nothing” — the classic ROS-in-Docker symptom. Here’s why it happens, how two containers in this project share a network and a block of memory instead, and what that costs. How ROS 2 nodes find each other # ROS 2 has no master. Nodes discover each other through DDS — in Jazzy, eProsima Fast DDS by default — using the SPDP protocol: Each participant announces itself over UDP multicast (239.255.0.1) on ports derived from the domain ID \(d\): discovery multicast on \(7400 + 250d\), unicast on \(7410 + 250d + 2p\) for participant \(p\). For domain 42: UDP 17900 and 17910+. Peers exchange their topic endpoints; matching publishers and subscribers connect. Data flows over the best transport both support — shared memory when they share a host and /dev/shm, UDP otherwise. Why Docker’s default network breaks it # Compose attaches services to a bridge network — a private NATed subnet. Multicast isn’t reliably routed across it, and machines on your LAN can’t reach container IPs at all.