Awesome GitHub RepositoriesData Science Tools

MindsDB is an AI-native database engine that treats machine learning models and autonomous agents as virtual tables. By mapping external data sources, predictive models, and third-party services directly into the database schema, it enables users to perform inference, data retrieval, and complex orchestration using standard SQL syntax. The platform distinguishes itself through an autonomous agent orchestrator that executes iterative reasoning loops, allowing agents to plan data access and synthesize natural language responses from connected knowledge bases. It functions as a federated data ga

Comet LLM is an observability platform and evaluation framework designed for large language model applications and agentic workflows. It functions as a system for tracing, monitoring, and debugging execution flows while providing tools for prompt optimization and the enforcement of AI safety guardrails. The platform distinguishes itself through a combination of model-based scoring and heuristic metrics to quantify output quality and detect hallucinations. It includes a dedicated prompt and agent optimizer with an interactive playground for refining templates and tool configurations. For retri

DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models. It functions as a system for managing large data artifacts by storing lightweight metadata in version control while keeping the actual binaries in a separate cache. The project serves as an experiment tracker and remote storage synchronizer, enabling the execution and comparison of machine learning iterations based on hyperparameters and performance metrics. It provides a bridge for pushing and pulling these large data artifacts between local environments and cloud or on-premi

Albumentations is a computer vision image augmentation library designed to increase training data diversity for deep learning models. It provides a toolset for applying geometric and color transformations to images and annotations, including a specialized collection of 3D operations for volumetric data used in medical and scientific imaging. The library functions as an image mask and bounding box transformer, automatically updating masks, bounding boxes, and keypoints when images undergo geometric changes. This ensures that spatial alterations remain synchronized across images and their assoc

Cleanlab is a data-centric AI library and toolkit designed to improve machine learning model performance by detecting label errors and increasing overall dataset quality. It implements a confident learning framework that iteratively refines label noise estimates by comparing model predictions with estimated label probabilities to identify mislabeled examples. The project provides specialized utilities for active learning optimization, allowing for the selection of the most impactful examples for labeling or re-labeling. It also includes an outlier detection tool to identify atypical data poin

AutoGluon is an automated machine learning framework designed to optimize model selection and hyperparameter tuning across tabular, text, image, and time series data. It functions as an ensemble learning library and a tabular data prediction engine, aiming to build high-accuracy predictive models without manual algorithm selection. The framework integrates multimodal machine learning pipelines that combine disparate data types into a single representation using specialized encoders. It also includes a probabilistic time series forecaster that fits multiple statistical and deep learning models

Gridstudio is a web-based data science integrated development environment that combines a programmatic spreadsheet interface with an interactive Python environment. It functions as a system for organizing and deploying isolated data workspaces to handle data science tasks and storage. The platform merges spreadsheet data management with an execution engine for formulas and Python code, allowing for programmatic spreadsheet manipulation. It enables users to run interactive scripts and terminal sessions to clean, transform, and manage datasets within a browser. The environment supports Linux s

Featuretools is an automated feature engineering library and data transformation framework written in Python. It automatically generates machine learning feature vectors from multi-table datasets by applying synthesis patterns to relational and timestamped data. The system functions as a distributed feature synthesis engine, allowing the process of creating feature vectors to scale across multiple cores or clusters to handle large-scale datasets. The library supports the synthesis of multi-table datasets, time series feature generation, and the creation of custom machine learning primitives

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

CML ist ein Pipeline-Automatisierungstool zum Trainieren und Evaluieren von Machine-Learning-Modellen und fungiert als CI/CD-System für Machine Learning. Es dient als Cloud-Compute-Orchestrator und Git-basierter Workflow-Manager, der Machine-Learning-Trainingszyklen durch Branch-Management, automatisierte Commits und integriertes Reporting automatisiert. Das Projekt zeichnet sich dadurch aus, dass es ephemere Cloud-Instanzen oder Kubernetes-Nodes bereitstellt, um spezialisierte Hardware für rechenintensive Aufgaben zur Verfügung zu stellen. Es verwaltet zudem Remote-Compute-Runner, was die Anbindung selbstgehosteter GPU-Cluster oder On-Premise-Maschinen zur Ausführung containerisierter Machine-Learning-Workflows ermöglicht. Das System deckt ein breites Spektrum an Funktionen ab, einschließlich ML-Experiment-Tracking, bei dem Leistungsmetriken und Visualisierungen direkt in Pull Requests der Versionsverwaltung gepostet werden. Es handhabt die ML-Pipeline-Automatisierung vom initialen Datenimport und der Versionierung bis hin zur Generierung formatierter Workflow-Berichte und externer Visualisierungslinks. Das Tool bietet zusätzlichen Nutzen für das Infrastruktur-Management durch SSH-basiertes Remote-Debugging und die Möglichkeit, unterbrochene Jobs fortzusetzen.

Dieses Projekt ist eine AWS-Pandas-Integrationsbibliothek und ein Daten-Pipeline-Framework, das entwickelt wurde, um die Bewegung und Transformation von Daten zwischen lokalem Speicher und AWS-Speicher- und Analysediensten zu vereinfachen. Es fungiert als Cloud-Data-Lake-Toolkit und Storage-File-Manager, der es Nutzern ermöglicht, strukturierte Daten über verschiedene Cloud-Umgebungen hinweg zu lesen, zu schreiben und zu transformieren. Die Bibliothek zeichnet sich als verteilter Compute-Orchestrator aus, der Cluster in Umgebungen wie EMR verwalten kann, um Datensätze zu verarbeiten, die die Speichergrenzen einer einzelnen Maschine überschreiten. Sie bietet zudem spezialisierte Funktionen zur Verwaltung von Vektor-Indizes und zur Durchführung von Ähnlichkeitssuchen innerhalb von Cloud-Storage-Buckets. Die breiteren Funktionen umfassen Cloud-Datenbank-ETL für Dienste wie DynamoDB, RDS und Timestream sowie Cloud-Data-Catalog-Management via AWS Glue. Sie unterstützt serverlose Datenanalyse durch Athena und Redshift und bietet Utilities zur Verwaltung von S3-Objekten, zur Indexierung von Dokumenten in OpenSearch und zur Analyse von CloudWatch-Logs.

Rodeo is an interactive Python notebook environment and integrated development environment designed for data science. It provides a workspace for combining executable code, rich text, and data visualizations within a single document to manage the lifecycle of research scripts. The platform facilitates data science workflow management, covering the process from initial data exploration to final model execution. It supports the development of Python scripting environments tailored for data analysis, modeling, and iterative hypothesis testing. The system utilizes a cell-based document structure

🛠 All-in-one web-based IDE specialized for machine learning and data science.

Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.

A GUI for Pandas DataFrames

Julia kernel for Jupyter

Karate Club: An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs (CIKM 2020)

🏕️ Reproducible development environment for humans and agents

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Hopsworks - Data-Intensive AI platform with a Feature Store