dmlc/xgboost
Xgboost
XGBoost is a distributed machine learning library for implementing scalable gradient boosting decision trees used for regression, classification, and ranking. It functions as a predictive model framework and a cross-language toolkit, providing a core implementation with native bindings for Python, R, Java, Scala, and C++.
The system is designed as a GPU-accelerated library that utilizes CUDA and NCCL to speed up the training of decision tree ensembles. It operates as a distributed framework capable of scaling training and prediction across multi-node clusters and GPU environments to process massive datasets.
The library covers a wide range of modeling tasks, including random forests, learning to rank, and survival or quantile regression. It includes capabilities for model optimization through custom loss functions, model interpretability via SHAP value computation and feature importance analysis, and data management techniques like external memory handling for datasets that exceed available system memory.
Features
- Gradient Boosting Libraries - Provides a scalable framework for implementing gradient boosting decision tree ensembles for regression, classification, and ranking.
- Cross-Language ML Toolkits - Offers a core C++ implementation with native bindings for Python, R, Java, Scala, and C++.
- Histogram-Based Learning - Uses histogram-based binning of continuous features to accelerate the search for optimal split points.
- Distributed Training - Scales model training across multi-node clusters and GPUs to process datasets with billions of examples.
- Distributed Training Frameworks - Functions as a distributed framework for scaling machine learning model training across multi-node clusters and GPU environments.
- Distributed Gradient Synchronization - Synchronizes gradients and model updates across compute nodes using a collective all-reduce communication pattern.
- GPU-Accelerated Training - Utilizes CUDA and NCCL to accelerate the construction of decision tree histograms and splits on GPUs.
- GPU Training Accelerators - Utilizes CUDA and NCCL to accelerate the training of decision tree ensembles on GPU hardware.
- Tabular Predictive Models - Provides a framework for building predictive models on structured tabular data using boosted trees and random forests.
- Distributed GPU Task Runners - Utilizes CUDA and NCCL to accelerate model processing through distributed GPU support across clusters.
- GPU & Performance - Executes training and prediction algorithms on CUDA-capable hardware to reduce total processing time.
- Feature Importance Attribution - Identifies which input variables have the greatest impact on predictions to improve model interpretability.
- Large Scale Training - Provides techniques for training models on massive datasets and distributed infrastructure.
- Learning to Rank Frameworks - Ranks a set of items based on relevance using specialized strategies and metrics.
- Custom Loss Functions - Supports the definition of custom gradient and hessian calculations for specialized objective functions.
- Early Stopping Monitors - Monitors model performance during training and terminates the process when validation metrics stop improving to prevent overfitting.
- Model Interpretability - Explains the output of decision tree ensembles through feature attribution.
- Model Interpretability Tools - Implements tools and techniques for explaining and interpreting the predictions of complex tree ensembles.
- Model Serialization - Provides binary model serialization that ensures consistent reproduction across different hardware and operating systems.
- Cross-Validation Utilities - Provides utilities for splitting datasets and performing iterative cross-validation to tune hyperparameters.
- Multi-GPU Training Utilities - Integrates with Dask to distribute training and prediction workloads across multi-node and multi-GPU clusters.
- Random Forest Ensembles - Implements random forest classifiers and regressors using a compatible API for ensemble learning.
- SHAP Value Computations - LightGBM computes feature importance scores for individual examples to explain model outputs using SHAP.
- Survival Analysis - Implements accelerated failure time objectives for modeling the time until a specific event occurs.
- Training Memory Management - Manages memory for large-scale training by loading data batches from disk when datasets exceed available RAM.
- Native Categorical Splitting - Processes categorical features natively via partition-based splitting to improve efficiency and accuracy.
- Sparsity-Aware Splitting - Ignores missing values during tree node splitting by learning a default direction for each missing feature.
- External Memory Block Compression - Handles massive datasets by storing data in compressed on-disk blocks and loading them as needed.
- JSON Serializers - Saves and loads models using a locale-independent JSON format for cross-platform reproducibility.
- C++ Bindings - Implements a high-performance C++ core with native bindings for Python, R, and Java for cross-language interoperability.
- Language Bindings - Provides high-performance shared libraries and bindings to enable use across Python, R, and Java.
- AI & Machine Learning - Scalable and distributed gradient boosting library
- Decision Tree Models - Optimized gradient boosting library.
- Deep Learning Frameworks - Provides scalable and distributed gradient boosting libraries.
- General Machine Learning - Optimized, parallelized gradient boosting library.
- Gradient Boosting - Library for extreme gradient boosting on decision trees.
- Gradient Boosting Research - Scalable system for tree boosting and feature selection.
- Machine Learning - Optimized gradient boosting library.
- Framework-uri de Machine Learning - Scalable and efficient gradient boosting library for structured data.
- Survival Analysis - Gradient boosting with survival analysis objectives.
- Computation and Optimization - Optimized distributed gradient boosting library.
