yeemachine/kalidokit
Kalidokit
Kalidokit is a web-based motion capture tool that transforms real-time webcam video into 3D character animation data. It functions as a blendshape and kinematics calculator, converting facial, hand, and body tracking data from Mediapipe and TensorFlow.js into blendshape weights and euler rotations for driving digital puppets and avatars.
The tool solves face landmarks to derive head rotation, eye blinks, mouth shapes, and brow values for rigging, while hand landmarks are converted into finger joint rotations and body keypoints into per-joint euler rotations for full-body animation. It includes specialized capabilities such as stabilizing eye blinks based on head rotation angle and processing landmark data frame-by-frame without buffering for real-time performance.
Kalidokit supports facial animation rigging, full-body pose estimation, hand gesture tracking, and real-time avatar control from a single webcam feed. The project provides a landmark normalization pipeline and quaternion-based joint rotation computation to ensure smooth skeletal animation across face, hands, and body.
Features
- Web-Based Capture Tools - Provides browser-driven motion capture from webcam landmark data for real-time 3D character animation.
- Webcam Motion Capture Pipelines - Converts real-time webcam video into 3D character animation data using face, hand, and body tracking.
- Head-Pose Euler Decompositions - Derives head rotation, tilt, and lean angles from face landmarks using perspective-n-point algorithms.
- Landmark-to-Blendshape Calculators - Converts Mediapipe/TensorFlow.js landmarks into blendshape values and joint rotations for 3D character rigging.
- Landmark-to-Blendshape Mappers - Converts raw landmark coordinates into normalized blendshape values using geometric calculations and mapping tables.
- Facial Animation - Drives blendshape facial rigs with head rotation, eye blinks, mouth shapes, and brow movements from landmark data.
- Face Landmark Solvers - Converts 468 face landmark points into head rotation, eye blink, mouth shape, and brow values for rigging.
- Eye-Blink Wink Detectors - Ships a dedicated eye-blink and wink detector that analyzes eyelid landmarks relative to head orientation.
- Hand - Extracts finger joint rotations from hand landmarks for real-time hand animation or gesture recognition.
- Landmark Solvers - Converts 21 hand landmark points into finger joint rotations for left or right hand rigging.
- Whole-Body - Transforms body keypoints into per-joint euler rotations for upper and lower body character rigging.
- Mediapipe Landmark Solvers - Computes head rotation, eye blinks, mouth shapes, and finger joint angles from Mediapipe landmark outputs.
- Pose-Driven Character Rigging - Transforms facial, hand, and body tracking data into blendshape weights and euler rotations for driving digital puppets and avatars.
- Pose Landmark Solvers - Transforms 33 body keypoints into per-joint euler rotations for upper and lower body animation.
- Finger Kinematics Solvers - Implements finger kinematics solvers that compute joint angles from hand landmark positions for rigging.
- Real-Time Motion-Driven Avatars - Controls a 3D avatar's face, hands, and body in real time using a single webcam feed.
- TensorFlow.js Motion Pipelines - Processes real-time webcam landmark data from TensorFlow.js models into rig-ready animation parameters for avatars.
- Stabilized Blink Transitions - Smooths left and right eye blink delays and wink transitions based on head rotation angle.
- Real-Time Landmark Stream Processors - Processes landmark data frame-by-frame from Mediapipe/TensorFlow.js outputs without buffering for real-time performance.
- Joint Rotation Quaternion Solvers - Computes joint rotations as quaternions from landmark positions for smooth skeletal animation.
- Landmark Normalization Pipelines - Provides a landmark normalization pipeline that scales and orients coordinates for consistent processing.
