vladmandic/human
Human
Human is a TensorFlow.js computer vision library used for face, body, and hand tracking within the browser or Node.js. It provides a framework for human pose and gesture tracking, facial recognition, and biometric liveness detection to verify a live human presence.
The project distinguishes itself through a full suite of identity and motion tools, including a facial recognition framework that generates embeddings for similarity matching and a background segmenter for separating humans from their environment. It incorporates a liveness detector to prevent spoofing during facial analysis.
The library covers a broad range of computer vision capabilities, including 3D body skeleton and hand landmark detection, iris movement monitoring, and facial attribute prediction for age, gender, and emotion. It includes infrastructure for media input processing, GPU acceleration, and the rendering of detection results as 2D overlays or 3D meshes on a canvas.
The project can be deployed as a systemd daemon on Linux and includes tooling for hosting local development servers via HTTP or HTTPS.
Features
- Browser-Based Model Inference - Runs machine learning models in the browser or Node.js using GPU acceleration via WebGL or WebGPU.
- Facial Recognition - Generates face embeddings and compares descriptors to identify individuals against a database.
- Computer Vision Libraries - Provides a JavaScript library for face, body, and hand tracking using TensorFlow.js models in the browser or Node.js.
- Object Detection - Identifies and locates non-human objects within images or video streams using machine learning models.
- Biometric Descriptors - Generates numerical biometric descriptors as facial embeddings to enable identity verification via similarity matching.
- Face Detection - Locates faces within images or video streams to enable targeted analysis of facial features.
- Liveness Detectors - Analyzes facial attributes and movements to verify a live human presence and prevent spoofing.
- Face Embeddings - Extracts unique numerical descriptors from face images to represent facial features for biometric comparison.
- GPU-Accelerated Inference - Accelerates machine learning computations by utilizing GPU drivers and graphics APIs for faster inference.
- Whole-Body - Simultaneously detects 3D landmarks for the body, face, and hands to recognize gestures and posture.
- State-Based Landmark Tracking - Maps 3D keypoints for the head, body, and hands to analyze gestures and orientation across video frames.
- Full Body Capture - Performs comprehensive 3D tracking of the head, face, eyes, body, and hands for virtual model animation.
- Background Segmentation - Separates humans from their backgrounds to produce foreground-only or alpha-channel images.
- Facial Similarity Matching - Compares an input image against a set of known faces to retrieve the most similar person.
- Video Input Processing - Provides real-time capture and streaming of live video frames from devices for AI-based visual analysis.
- Human Pose Detections - Identifies human faces, body skeletons, and hand gestures in real-time video frames.
- Face Tracking - Tracks 3D face rotation and iris movements to determine gaze and head orientation.
- Detection-to-Track Association - Implements logic to associate detected body parts and features with specific individuals for consistent tracking.
- Capture Readiness Validation - Analyzes camera input against required tests to ensure a face is properly positioned before capturing.
- Gesture Recognition Libraries - Processes landmark coordinates to identify and classify specific human gestures in real-time.
- Detection Visualization - Renders detected faces, bodies, hands, and objects as visual overlays on a canvas for verification.
- Facial Analysis Toolkits - Predicts demographic and emotional attributes, such as age, gender, and emotion, from facial features.
- Hardware-Agnostic Inference Switching - Optimizes inference speed and latency by automatically switching between CPU and GPU runtimes based on availability.
- Iris Tracking - Provides iris movement monitoring to determine gaze direction and eye orientation.
- Model Execution Environments - Provides a CPU-based execution environment for running inference in systems lacking dedicated graphics hardware.
- Model Output Visualizers - Renders detection results as points, outlines, or 3D meshes to visually represent features.
- Temporal Prediction Smoothing - Uses temporal prediction smoothing to reduce coordinate jitter and ensure fluid movement of detected landmarks across frames.
- Inference Optimizers - Optimizes inference speed by selecting the fastest available hardware backend and implementing model caching.
- Canvas Rendering - Provides the ability to render 3D meshes and detection overlays directly onto an HTML5 canvas.
- Cross-Platform Media Frameworks - Analyzes images and videos across browser and server runtimes using native methods or external tools.
- Media Analysis - Analyzes visual data from images, canvases, webcams, and various video streaming formats.
- Temporal Interpolation - Generates smooth transitions between consecutive detection data points to reduce visual jitter across frames.
- Coordinate Smoothing - Blends consecutive detection results to eliminate jumping and ensure fluid visual updates of tracked landmarks.
- Asynchronous Processing Pipelines - Implements asynchronous pipelines to handle heavy model execution in background threads, maintaining browser responsiveness.
- Main Thread Offloading - Offloads computationally expensive vision analysis to background web workers to prevent blocking the main UI thread.
