Awesome GitHub RepositoriesData Analysis and Processing

Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized

Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external

NumPy is a foundational library for scientific computing in Python, providing a comprehensive framework for managing and manipulating large-scale numerical information. It centers on high-performance multidimensional array objects that serve as the primary data structure for complex mathematical operations and data analysis workflows. The library distinguishes itself through specialized mechanisms for handling multidimensional data, including advanced indexing, slicing, and broadcasting techniques that allow for efficient operations across arrays of varying shapes. It utilizes strided metadat

Matplotlib is a Python data visualization library and 2D plotting engine used to generate publication-quality figures and charts from numerical data. It serves as a numerical graphics library and data visualization toolkit for mapping data to visual elements. The library provides capabilities for producing static, animated, and interactive visualizations. This includes creating high-resolution figures for professional documents, generating moving graphics to illustrate data evolution over time, and building dynamic plots for interactive data exploration. The toolkit supports scientific plott

Bokeh is a Python data visualization library and interactive plotting framework used to create high-performance graphics and data dashboards that render in web browsers. It serves as a tool for generating standalone HTML documents, embedded components for digital notebooks, and full-stack web applications powered by a Python backend. The project distinguishes itself through its ability to handle large or streaming datasets while maintaining smooth interactivity. It enables linked brushing across multiple views, allowing data selected in one plot to automatically highlight corresponding data i

Plotly.py is a comprehensive framework for building production-ready data applications and interactive dashboards directly from Python code. It functions as both a high-performance visualization library for browser-based charts and a full-stack tool for transforming analytical scripts into responsive, web-based interfaces. By abstracting away the need for manual HTML or JavaScript, it allows developers to define complex layouts and functional logic using modular, reusable components. The framework distinguishes itself through a robust architecture that handles event orchestration and state sy

NetworkX is a Python library designed for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. It provides a comprehensive framework for modeling relationships between entities as graphs, directed graphs, or multigraphs, allowing users to attach arbitrary metadata and properties to nodes and edges. The library distinguishes itself through a modular architecture that decouples graph analysis logic from data storage, utilizing nested dictionaries and adjacency lists to manage topology. It features a pluggable backend system that delegates computat

pyecharts is a Python visualization library and wrapper for the Echarts JavaScript engine. It translates Python data and configurations into JSON specifications to generate interactive web-based charts and graphs. The library provides specialized capabilities for geographic data mapping using a comprehensive library of map assets to visualize spatial information. It also includes utilities to capture rasterized snapshots of rendered web visualizations for export as static image files. The tool supports rendering interactive plots directly within data science notebook environments and exporti

Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.

SciPy is a scientific computing library for Python that provides a comprehensive collection of mathematical algorithms and numerical tools for research and engineering. It functions as a high-performance numerical analysis framework, bridging high-level Python code with compiled C and Fortran routines to execute complex computations at hardware speeds. The library is built upon array-based data structures that utilize strided memory layouts to enable efficient data manipulation and slicing. By employing vectorized operation dispatch and linking to optimized hardware-specific linear algebra li

Dask este un framework de calcul paralel și un scheduler de sarcini distribuit conceput pentru a scala fluxurile de lucru de știința datelor în Python de la mașini individuale la clustere mari. Acesta funcționează ca un manager de resurse de cluster care orchestrează logica computațională prin reprezentarea sarcinilor și a dependențelor acestora sub formă de grafuri aciclice direcționate. Această arhitectură permite sistemului să automatizeze distribuția sarcinilor de lucru pe hardware-ul disponibil, gestionând în același timp cerințe complexe de execuție. Proiectul se distinge printr-un motor de evaluare leneșă (lazy) care amână operațiunile pe date până când sunt solicitate explicit, permițând optimizarea globală a grafului și alocarea eficientă a resurselor. Acesta încorporează „spilling” de date conștient de memorie pentru a preveni blocarea sistemului la procesarea seturilor de date care depășesc memoria disponibilă și utilizează fuziunea grafului de sarcini pentru a combina secvențe de operațiuni în pași de execuție unici, minimizând overhead-ul de programare și comunicarea între noduri. Platforma oferă o suprafață cuprinzătoare de capabilități pentru analiza datelor la scară largă, inclusiv suport pentru învățare automată distribuită, integrare cu calcul de înaltă performanță și procesare paralelă a datelor. Oferă instrumente extinse pentru gestionarea ciclului de viață al clusterului, profilarea performanței și monitorizarea în timp real a execuției sarcinilor. Utilizatorii pot implementa aceste medii pe diverse infrastructuri, inclusiv hardware local, furnizori de cloud, sisteme containerizate și clustere de calcul de înaltă performanță.

This library provides a diagnostic toolkit for automated data profiling and exploratory analysis. It generates comprehensive statistical summaries and visual reports for tabular datasets, enabling users to identify distribution patterns, missing values, and quality anomalies through a unified interface. The project distinguishes itself by offering differential analysis, which allows for the comparison of two dataset versions to track structural and statistical changes over time. It supports large-scale data processing through lazy evaluation and provides interactive widgets that embed directl

This project is a data profiling and exploratory data analysis tool designed to generate automated quality reports for Pandas and Spark dataframes. It serves as a system for computing descriptive statistics, identifying correlations, and analyzing univariate and multivariate data patterns. The tool provides specialized capabilities for comparing different versions of datasets to identify changes in data quality and distributions. It includes a dedicated profiler for time-dependent data to extract statistical information such as seasonality and auto-correlation. The software covers a broad an

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in

Kedro is a data science pipeline framework and orchestration tool designed to build reproducible and modular data engineering workflows. It functions as an MLOps project template and Python data workflow tool that enforces software engineering best practices to move projects from prototype to production. The system distinguishes itself through a centralized data catalog manager that abstracts data access and versioning across various file formats and cloud storage systems. It further separates processing logic from data access via a lazy-loading data registry and provides a standardized proje

Altair is a declarative data visualization library for Python based on the Vega-Lite grammar. It allows users to create statistical visualizations by mapping data fields to visual properties rather than writing imperative drawing code. The library focuses on interactive charting through a system of linked selections and filters that update multiple visualizations based on user input. It renders charts as JSON and HTML for display in web browsers and interactive notebooks. The project covers statistical data analysis and interactive data exploration, providing capabilities to export visuals a

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

Polar is a digital product monetization engine and subscription management system. It serves as a merchant of record platform that handles global sales tax and VAT compliance, providing the infrastructure for selling subscriptions and one-time digital goods via hosted checkouts and embedded payment flows. The project functions as an entitlement and access manager, automating the granting and restriction of digital benefits, license keys, and third-party platform roles. It includes a dedicated usage-based billing infrastructure that tracks customer activity through meters to apply aggregation

Mage AI este un orchestrator de pipeline-uri de date bazat pe Python și un mediu de dezvoltare integrat (IDE) pentru date, self-hosted. Este conceput pentru construirea, programarea și monitorizarea workflow-urilor de date folosind un design de pipeline bazat pe blocuri și o interfață de notebook interactivă. Platforma se distinge prin integrarea capabilităților de AI generativ, permițând utilizatorilor să conecteze furnizori de modele de limbaj mari (LLM) prin API pentru a încorpora inteligența artificială în fluxurile de date automatizate. De asemenea, funcționează ca un procesor de date Apache Spark, gestionând kernel-urile și infrastructura necesară pentru analize de mare volum și procesarea datelor la scară largă. Sistemul acoperă o gamă largă de capabilități de inginerie a datelor, inclusiv automatizarea workflow-urilor ETL, gestionarea modelelor dbt și descoperirea fluxurilor de date. Oferă instrumente pentru integrarea controlului versiunilor prin Git, deployment containerizat și controlul accesului bazat pe roluri pentru a gestiona pipeline-urile în medii de dezvoltare și producție. Monitorizarea este gestionată prin telemetria performanței sistemului și debugging-ul execuției pipeline-urilor.

Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin