Two ways for a six-axis robot to respond: interpret the room through a vision-language model, or mirror a person through hand tracking. One webcam turns perception into live, expressive motion.
In VLM mode, Gemini reads energy, mood, presence, urgency, and gestures, then selects a motion behavior. In tracking mode, MediaPipe maps a person's hand directly to the arm's position and pitch.
The core ideas are semantic perception, embodied interaction, organic motion, and safe real-time control. Both modes share the same Cartesian motion pipeline, filtering, workspace limits, and recovery logic.
// interaction modes
Two control modes running through the same motion and safety system.
MediaPipe hand tracking — live hand landmarks drive the robot's position and pitchGemini VLM interaction — scene context and gestures become higher-level motion behaviors
// 02 · process · 1 image
VLM interaction mode — scene parameters, gesture state, arm telemetry, and the live camera feed in one debugging dashboard
