jeeliz/jeelizFaceFilter
JeelizFaceFilter
jeelizFaceFilter is a browser-based computer vision engine and WebGL face tracking library designed for AR filters and real-time facial movement tracking. It functions as a neural network face detector that identifies multiple faces and monitors mouth movements and rotation within a web browser.
The system distinguishes itself through a model-swappable detection pipeline, allowing the exchange of neural network weights to balance accuracy and performance across different camera angles and devices. It features real-time lighting synchronization to match the illumination of 3D overlays with the user's environment and generative AI capabilities to create 3D product models from images.
The library covers a broad range of capabilities, including GPU-accelerated rendering for virtual try-on experiences, coordinate-based asset mapping for 3D content integration, and execution state control to manage GPU resource consumption. It also provides tools for video input configuration, including camera resolution and device selection.
Features
- Neural Face Renderers - Renders lifelike 3D models onto detected faces using synchronized lighting for seamless integration.
- Augmented Reality Frameworks - Renders 3D objects and virtual assets onto a user's face through a web browser.
- Face Detection - Identifies and tracks several unique faces simultaneously within a single video stream.
- Face Tracking - Monitors face rotation and mouth opening in real time to provide coordinates for AR overlays.
- WebGL - Uses WebGL and neural networks to track facial movement and rotation in real time within a browser.
- Face Detection - Provides a configurable detection system that allows swapping models to balance performance and accuracy.
- Web-Based Computer Vision - Implements client-side face detection and movement monitoring directly within a web browser.
- Neural Networks - Uses machine learning models to locate faces and extract landmarks from a live video stream in real time.
- Digital Face Filters - Overlays 3D models and virtual assets onto detected faces with synchronized lighting and coordinates.
- GPU-Accelerated Media Rendering - Processes 3D overlays and visual effects using the GPU to maintain high frame rates on the web.
- Environmental Lighting Estimation - Calculates light source direction and intensity from video input to apply matching shaders to 3D objects.
- Real-Time Motion Tracking - Monitors facial rotation and mouth movements in a video stream for interactive digital overlays.
- Dynamic Lighting Synchronization - Matches the lighting of 3D overlays to the user's face in real time for realistic visual integration.
- Facial Landmark Mapping - Maps detected facial landmarks to a 3D coordinate system to align virtual objects with facial movements.
- 3D Asset Generators - Automatically creates detailed 3D product models from images and product pages using generative AI.
- Inference Performance Optimization - Adjusts face detection models to balance processing speed and accuracy based on camera angles or device performance.
- Model Weight Swapping - Allows switching between different neural network weights to optimize for specific camera angles or device performance.
- 3D Asset Integrations - Enables connecting facial tracking data to external engines or web APIs to render virtual objects onto faces.
- Virtual Try-On Interfaces - Allows users to see how 3D product models look on their face using real-time tracking and lighting.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
