A webcam, some vector geometry, and a hand that moves
Everything in this series in one read: how a $20 webcam ends up driving a 15-DOF CAD model in real time, why every step is deliberately explicit rather than learned, and what broke along the way. You hold your hand up to a laptop camera. On the other half of the screen, a robotic hand — designed in CAD, never manufactured — closes its fingers at the same moment yours do.
There is no glove, no marker, no depth sensor. Just an RGB webcam, two small neural networks, about forty lines of vector geometry, and a middleware stack that thinks it is talking to a real robot.
All of it is open source under AGPL-3.0 and archived with a DOI: 10.5281/zenodo.22658556.
flowchart LR A["📷 Webcam
/dev/video0"] --> B["BlazePalm
palm detector"] B --> C["Landmark regressor
21 × (x, y, z)"] C --> D["Dot-product geometry
15 interior angles"] D --> E["Normalize → flexion
0.0 straight · 1.0 curled"] E --> F["Lerp onto the URDF's
mechanical limits"] F --> G["/joint_states"] G --> H["robot_state_publisher
→ /tf"] H --> I["🖥️ RViz digital twin"] The rule that shaped the build # There is an easier version of this project. Collect a few thousand frames of a hand next to the corresponding CAD poses, train a network to map one to the other, and let gradient descent work out the relationship.