roboflow/rf-detr
Rf Detr
RF-DETR is a Python library for training and deploying object detection, instance segmentation, and keypoint detection models built on a vision transformer architecture. It provides a unified command-line interface and Python API for the full workflow, from fine-tuning pretrained checkpoints on custom datasets to running inference on images, video files, and live camera streams.
The project supports training on datasets in COCO or YOLO format, with automatic format detection and configurable augmentation pipelines. Models can be exported to ONNX, TFLite, or TensorRT for deployment across edge hardware, mobile devices, and serverless APIs. Training includes built-in experiment tracking with TensorBoard, Weights and Biases, MLflow, and ClearML, along with multi-GPU support, early stopping, and automatic checkpoint selection based on validation mAP.
Inference capabilities cover batch processing, real-time detection from webcams or RTSP streams, and per-instance segmentation masks. The library also provides tools for converting between dataset formats and caching model weights locally for faster repeated predictions.
Features
- Instance Segmentation Engines - Produces per-instance segmentation masks for every detected object using a single unified model API.
- Object Detection - Runs a pretrained vision transformer model to locate and classify objects with bounding boxes.
- Real-Time Object Detection - Runs transformer-based detection models on images and returns bounding boxes and class labels with low latency.
- Image Segmentation - Runs pretrained segmentation models on static images and returns detected objects with masks and labels.
- Custom Model Training - Fine-tunes a pre-trained detection model on a custom dataset with configurable parameters.
- Detection Model Training - Trains detection, segmentation, and keypoint models on custom datasets with automatic COCO or YOLO format detection.
- Vision Detection Model Training - Trains custom vision detection, segmentation, and keypoint models from pretrained checkpoints on user datasets.
- Image Inference Clients - Accepts images as file paths, URLs, PIL Images, or arrays and returns detections.
- Keypoint Detection - Ships a vision transformer model for detecting human keypoints in images.
- Model Fine-Tuning - Fine-tunes pretrained vision transformer models on custom datasets for detection, segmentation, and keypoint tasks.
- Checkpoint Loaders - Loads fine-tuned models from checkpoints to run predictions on images and videos.
- Model Checkpoints - Restores model weights from a checkpoint file, auto-detecting the correct variant.
- Pretrained Checkpoint Fine-Tuning - Starts training from COCO-pretrained weights across model sizes to accelerate convergence and improve accuracy.
- CLI Training Toolkits - Provides a unified CLI with fit, validate, test, and predict subcommands for managing training workflows.
- Segmentation Model Training - Configures and trains a model to predict per-pixel class labels for detected object regions.
- ONNX Model Exporters - Converts trained models to ONNX format for cross-platform inference with ONNX Runtime, OpenVINO, or TensorRT.
- Image Augmentation Transforms - Configures per-dataset augmentation pipelines with automatic bounding box and mask handling.
- Model Evaluation Metrics - Computes mAP, precision, recall, and F1 on a validation dataset after each epoch.
- Domain-Specific Augmentation Presets - Selects from curated augmentation presets optimized for dataset size or domain like aerial and industrial.
- Validation-Based Checkpoint Selection - Automatically saves and selects the best-performing model checkpoints based on validation mAP scores during training.
- Validation Evaluators - Evaluates model performance on a validation dataset and logs mAP and F1 scores.
- Training Checkpointers - Saves and resumes training states including model weights, optimizer state, and epoch number from checkpoints.
- Experiment Tracking - Logs training metrics to TensorBoard, Weights and Biases, ClearML, and MLflow for experiment monitoring.
- Model Export Formats - Converts trained detection models to ONNX and TFLite for production deployment.
- Video Object Segmentations - Processes frames from video files, webcams, or RTSP streams in real time with segmentation masks and labels.
- Augmentation Geometric Transforms - Registers custom geometric transforms that automatically update bounding boxes and masks during augmentation.
- Augmentation Pipeline Composers - Combines image augmentation transforms with probabilistic containers like OneOf, SomeOf, and Sequential.
