Awesome GitHub RepositoriesMulti-Source CSV Loading

TensorFlow.js is a JavaScript machine learning library used for training and deploying models in web browsers and server-side environments. It functions as a browser-based model trainer, a WebAssembly inference engine, and a WebGPU accelerated tensor library for low-level linear algebra. The project also includes a model converter to transform Python-based models into optimized formats for JavaScript execution. The library distinguishes itself through a pluggable backend architecture that allows mathematical operations to be executed via CPU, WebGL, or WebGPU. It supports the conversion of Py

Perspective is a columnar data analytics engine and high-performance visualization component powered by WebAssembly. It provides a system for analyzing and visualizing large or streaming datasets through interactive data grids and charts, utilizing a compiled binary to achieve near-native performance within the browser. The project distinguishes itself through a WebSocket-based data streaming interface and deep Apache Arrow integration, which minimize memory overhead when synchronizing tables between servers and clients. It acts as a remote query proxy capable of translating visualization con

Apache DataFusion is an extensible, columnar SQL query engine that runs embedded within a host application without requiring a separate server process. It processes data in columnar batches using Apache Arrow for memory-efficient analytics, and can scale analytic workloads across multiple nodes for parallel execution. The engine supports both SQL and DataFrame queries through a modular, streaming architecture that allows custom operators, data sources, functions, and optimizer rules. The engine distinguishes itself through its modular extension framework, which enables building custom query e

AlaSQL is a JavaScript SQL database engine that allows for the filtering, grouping, and joining of in-memory object arrays and JSON data. It functions as an in-memory SQL database and client-side data processor, enabling the execution of SQL statements against JavaScript arrays and external data sources in both browser and server environments. The project serves as a universal data query tool capable of performing relational joins across diverse sources, such as merging Google Spreadsheets, SQLite files, and remote APIs into a single result set. It also acts as an IndexedDB SQL wrapper, allow

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

This repository is the official documentation for TensorFlow, a machine learning framework. It provides comprehensive guides, tutorials, and API references for building, training, and deploying machine learning models. The documentation covers the full lifecycle of machine learning projects, from constructing data pipelines and building neural networks with high-level APIs to customizing training loops and deploying trained models in production, on edge devices, or in browsers. The documentation includes step-by-step tutorials for a range of tasks, including reinforcement learning, ranking mo

pgloader is a command-line tool that automates the migration of data and schema from various source databases and file formats into PostgreSQL. It combines schema discovery, parallel data pipelines, and type casting into a single, declarative workflow, using PostgreSQL's COPY protocol for high-throughput bulk loading. The tool distinguishes itself by compiling a dedicated command language into concurrent reader-writer pipelines that handle schema introspection, data transformation, and error-resilient batch processing. It supports migrating entire databases from MySQL, MS SQL, SQLite, and Pos

PlotJuggler is an interactive time series visualization tool that loads, streams, and renders large datasets using hardware-accelerated OpenGL graphics. It functions as a multi-format data loader, supporting file formats such as CSV, ULog, and ROS bags, and also serves as a live data stream viewer that subscribes to real-time sources via MQTT, WebSockets, ZeroMQ, and UDP. The tool distinguishes itself through a plugin-based extensibility platform that allows users to add custom data sources, file formats, and processing capabilities. It includes a Lua scripting engine for creating custom data

River 是一个用于在线机器学习的 Python 框架，旨在对流式数据进行模型训练和评估。它通过一次处理一个观测值来更新模型参数，从而实现增量学习，无需在内存中存储完整的训练数据集。 该库通过专门的概念漂移（Concept Drift）检测系统脱颖而出，该系统监控数据分布的变化以触发模型自适应。它还提供了一个渐进式验证框架，通过在训练前对样本进行测试来模拟实时部署。 该系统涵盖了广泛的流式处理功能，包括实时特征工程、时间序列预测和在线异常检测。它支持通过增量聚类和决策树进行无监督学习，以及用于模型选择的集成聚合和 Bandit 策略。 该项目包括从 CSV 文件和 API 等来源进行流式数据摄取的实用程序，以及用于计算运行统计信息和内存高效数据草图（Data Sketches）的工具。

Anomalib is a PyTorch-based library for visual anomaly detection, offering a modular framework, a comprehensive model zoo, and a benchmarking suite designed for industrial defect detection. It provides a wide range of algorithms—including generative, discriminative, teacher-student, and vision-language approaches—that support unsupervised, few-shot, and zero-shot settings. The library enables deployment through model export to ONNX and OpenVINO for edge devices, and includes a no-code web application for training and inference. It also features a command-line interface for orchestrating multi

NVIDIA DALI is a GPU-accelerated data loading and preprocessing library designed for deep learning workflows. It constructs high-performance data pipelines that offload decoding, augmentation, and normalization to the GPU, eliminating CPU bottlenecks in training and inference. The library reads data from multiple storage formats and streams it directly into GPU memory, with support for multi-GPU execution to scale throughput across large-scale workloads. DALI distinguishes itself by enabling data pipelines to be built once and executed across multiple deep learning frameworks without code cha

Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data

这是一个图形语法可视化库，用于通过将表格数据映射到视觉标记来构建图表。它作为一个 SVG 数据可视化工具和探索性数据分析 API，允许用户渲染复杂的可视化效果和地理地图。 该库具有一个 GeoJSON 地图渲染器，可将球坐标投影到二维像素空间，以及一个用于高效数据处理的 Apache Arrow 可视化接口。 其功能面涵盖通过分箱（binning）和分组进行数据转换、通过自动比例推断和配色方案应用进行视觉编码，以及生成小多重图（small multiples）。它支持在分层视图中渲染几何形状，并在服务器端环境中导出静态图像。

该项目是一个开源搜索数据索引和作为公共趋势档案提供的历史搜索趋势数据集合。它作为一个开放数据集，用于通过可下载文件分析全球模式和事件。 该仓库提供了一个匿名化和归一化搜索及媒体数据集的聚合索引。这些资源专为学术和专业分析而设计，允许研究不同地区和时间跨度的纵向趋势。 该数据支持全球搜索趋势分析、市场模式分析和公共利益研究。它实现了用于研究消费者兴趣、社会变迁和搜索行为的开放数据获取。

ExcelDataReader 是一个 C# 库，用于从 Microsoft Excel 电子表格和 CSV 文件中提取数据和元数据。它作为一个工作簿解析器，将电子表格内容转换为结构化数据集，以便进行程序化访问和迭代。 该项目包含一个专门的元数据提取器，用于检索单元格级别的详细信息，例如数字格式、样式、行高、列宽和合并单元格范围。它还提供了一个流处理器，用于解析具有可自定义编码和分隔符检测功能的纯文本 CSV 文件。 该库支持现代电子表格文件的 OpenXML 标准，并利用基于流的解析和基于游标的行迭代来遍历工作簿。这些功能允许将多工作表工作簿转换为关系数据表。

docetl is an AI-powered document ETL tool and map-reduce orchestrator designed to transform large collections of unstructured documents into structured, queryable tables using language models. It provides a declarative pipeline framework for extracting, cleaning, and transforming data from sources such as PDFs and text files into predefined schemas. The project distinguishes itself through a semantic data integration suite that enables joining datasets and resolving duplicate entities based on embedding-based similarity. It includes an interactive prompt playground for developing and optimizi

mcp-context-forge is a Model Context Protocol federation gateway that unifies diverse AI tool servers and APIs into a single consistent interface for discovery and execution. It acts as a centralized proxy that aggregates multiple servers and APIs, allowing AI agents to access and invoke a unified set of tools, prompts, and resources. The project distinguishes itself through a multi-protocol translation bridge that converts communication between standard I/O, SSE, gRPC, and REST to enable interoperability between disparate tool servers. It includes a comprehensive LLM evaluation framework for