22 个仓库
Programming components that provide sequential access to elements within a large data collection during processing.
Explore 22 awesome GitHub repositories matching data & databases · Data Iterators. Refine with filters or upvote what's useful.
Developer Roadmap 是一个社区驱动的平台,提供结构化的、基于图谱的软件工程学习路径。它作为一个综合知识仓库,将技术领域组织成可视化序列,以指导专业技能获取和职业成长。 该项目通过协作生态系统脱颖而出,使用户能够贡献路线图、策划行业最佳实践并维护个人职业档案。它集成了诊断评估框架来评估技术熟练度,帮助开发者识别知识缺口,并通过有针对性的学习序列为专业面试做准备。 除了核心映射能力外,该平台还提供实用的项目创意和交互式辅导,以巩固工程概念。它为社区提供了一个共享资源、跟踪技能进步和导航复杂技术领域的中心化空间。
Provides sequential access to elements within large data collections during processing.
Faceswap is a comprehensive framework for automated media manipulation and neural face synthesis. It provides a modular pipeline that manages the entire lifecycle of facial feature extraction, deep learning model training, and image conversion. By coordinating complex computer vision workflows, the system enables users to map facial identities between source and destination datasets while maintaining structural alignment and lighting consistency across video frames. The project distinguishes itself through a highly extensible plugin-based architecture that handles hardware-accelerated process
Serves as a base class for plugins to ingest and pass information through the extraction pipeline.
LevelDB is an embedded database library and persistent storage engine that provides a sorted key-value store. It uses a log-structured merge-tree architecture to map byte arrays to values, running directly within a process to provide storage without the need for a separate server process. The system is distinguished by its use of custom comparison functions to define key ordering, enabling efficient range scans and sequenced lookups. It ensures data reliability through atomic batch execution, consistent snapshot generation, and log-based recovery after failures. The engine covers broad capab
Provides sequential iterators for traversing stored entries in forward or backward order.
Immutable.js is a library of persistent data structures and a functional state management toolkit. It provides a collection of immutable objects and arrays that prevent direct mutation to ensure predictable state management in JavaScript applications. The library utilizes structural sharing to efficiently create new versions of data without full copying and implements lazy sequence processing to chain data transformations that execute only when values are requested. It also supports batch mutation processing, allowing multiple changes to be applied to a temporary mutable copy before returning
Implements memory-efficient lazy iterators that defer data transformations until values are explicitly requested.
Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca
Implements lazy, memory-efficient iterators to process large datasets on demand without loading them into physical memory.
This library is a collection of generic utilities for the Go programming language designed to simplify the manipulation of slices and maps. It provides a functional toolkit that enables developers to perform data transformations, such as filtering, mapping, and reducing, while maintaining strict type safety through the use of language-level generics. The project distinguishes itself by offering a dual approach to data processing that balances functional programming patterns with performance-oriented execution. It supports both immutable functional pipelines for predictable state transitions a
Provides a comprehensive toolkit for memory-efficient, lazy data traversal and deferred computation of large or infinite sequences in Go.
Excelize is a library for reading and writing spreadsheet files in the Office Open XML format. It provides a comprehensive suite of tools for programmatically creating, modifying, and analyzing workbooks, worksheets, and cell data, ensuring compatibility across various office software suites through structured XML serialization. The library distinguishes itself with a built-in formula calculation engine that evaluates complex mathematical and logical expressions directly against workbook data. It also features a memory-mapped streaming architecture, which allows for the efficient processing o
Emits data iteratively to maintain low memory usage during large-scale file processing.
Gensim is an unsupervised natural language processing toolkit designed for topic modeling, word embedding training, and the processing of large-scale text corpora. It provides a framework for discovering latent themes and semantic structures in text without the need for labeled data. The toolkit is distinguished by its ability to handle datasets that exceed system memory through iterator-based data streaming from disk. It also supports distributed model training, allowing complex modeling tasks to be executed across computer clusters. The library covers a broad range of analysis capabilities
Implements data iterators to stream large text collections from disk, avoiding memory exhaustion.
Home Assistant is a local home automation platform and server that acts as an IoT device orchestrator. It integrates diverse smart home hardware by wrapping third-party APIs into a standardized logic layer and stores all system state and historical statistics on local hardware to eliminate cloud dependencies. The system functions as a Matter IoT controller and an MQTT home automation bridge, allowing for local interoperability between different manufacturers. It features a state-based entity model and an internal event bus that decouple physical device logic from system automation. The platf
Converts lazy sequences produced by filters into static lists to enable counting and sorting.
EASTL is a C++ Standard Template Library implementation consisting of containers, iterators, and algorithms. It provides cross-platform data structures and a template-based algorithm library designed for use in resource-constrained game engine environments. The library focuses on game engine memory management, providing specialized utilities that ensure predictable memory allocation and high-performance access for real-time applications. These containers maintain consistent behavior across different operating systems and hardware platforms. The project covers high-performance C++ development
Provides standardized iterators for traversing diverse data collections without exposing underlying memory layouts.
This project is an educational resource and a collection of instructional materials for performing data manipulation and statistical analysis using Python. It provides a comprehensive set of guides and code examples for using the Pandas, NumPy, and Matplotlib libraries to analyze structured data. The resource includes a dedicated guide for reshaping, cleaning, and aggregating tabular data and time series via Pandas, alongside a reference for high-performance vectorized operations and linear algebra using NumPy. It also features tutorials for creating publication-quality charts, distribution p
Uses generators to produce sequences of values on demand, reducing memory consumption for large datasets.
Node.js is an open-source, cross-platform JavaScript runtime environment built on the V8 engine, designed for executing JavaScript code outside a web browser. It operates as a server-side JavaScript platform with an event-driven, non-blocking I/O architecture that enables building scalable network applications and web servers. The runtime integrates the CommonJS module system for synchronous module loading and the npm ecosystem for sharing and reusing packages. The platform provides comprehensive capabilities for web server development, including creating HTTP and HTTPS servers, managing HTTP
Supports processing streaming data with async iterators for chunk-by-chunk consumption without full buffering.
Lazy.js is a JavaScript library that implements a lazy evaluation model for processing collections and data streams. It defers all computation until iteration begins, building chains of transformations that execute only when values are consumed, avoiding intermediate arrays and buffering. The library wraps data sources into a uniform sequence interface, enabling operations like map and filter to be chained together without materializing intermediate results. The library extends lazy processing beyond simple collections to handle asynchronous data sources, DOM events, strings, and Node.js stre
Integrates with asynchronous data sources by yielding values at timed intervals or from streams without blocking.
r4ds 是一个数据科学课程和教育资源,专为精通 R 编程语言而设计。它为导入、整理、转换和可视化数据的端到端过程提供了结构化的学习路径。 该项目强调可重复的数据科学指南和全面的数据整理课程。它包括关于用于分层数据可视化的图形语法(grammar of graphics)的专业教程,以及使用 Quarto 创建的融合可执行代码与叙述性文本的技术出版物。 该材料涵盖了广泛的分析能力,包括来自不同来源的数据摄取、关系数据连接以及分类变量的管理。它还涉及数据清洗、数学建模以及多格式专业报告和演示文稿的生成。 该课程侧重于函数式编程和整洁数据(tidy data)原则的实际应用,以创建透明且可重复的分析。
Demonstrates how to apply a consistent set of actions across data collections using functional programming.
Toolz is a Python library that implements functional programming utilities for iterable transformation, dictionary manipulation, function composition, and lazy evaluation. It provides a set of pure functions designed to work with Python's built-in data structures, enabling concise and composable data processing workflows. What distinguishes toolz is its support for curried partial application, allowing functions to be incrementally applied and reused. It includes dictionary-centric operations that handle nested structures, and offers iterable chain transformers that combine mapping, filtering
Processes sequences on-demand using generators for memory-efficient handling of large data streams.
Slonik 是一个用于 Node.js 的类型安全 PostgreSQL 客户端,使用标记模板字面量(tagged template literals)来确保参数绑定并防止注入攻击。它提供了一个将应用连接到 PostgreSQL 的框架,并为查询和数据库模式提供自动类型检查。 该项目通过专门的 SQL 查询 Linter 脱颖而出,该 Linter 通过在开发过程中根据实时数据库模式验证代码,来检测无效列和类型不匹配。它还包括一个用于加载大数据集的高性能二进制批量数据插入器(使用原生二进制序列化),以及一个能够在主节点和副本节点之间进行动态查询路由的连接池管理器。 该库涵盖了广泛的数据库能力,包括原子事务管理、动态 SQL 查询构建,以及通过异步迭代流处理大数据集。它进一步提供了用于日志记录和基准测试的中间件拦截器、自定义类型解析,以及用于刷新数据库身份验证凭据的异步回调机制。
Provides memory-efficient processing of large database result sets using async iterable streams.
Ignite is a high-level training framework for PyTorch neural networks that serves as a training engine and deep learning lifecycle manager. It provides a structured system for organizing and automating training and evaluation loops, managing data iterators and triggering event handlers at specific milestones during the model training process. The project distinguishes itself through a comprehensive suite of tools for distributed training and model evaluation. It includes utilities for synchronizing gradients and coordinating collective communication across multiple GPUs or nodes, as well as a
Controls finite or infinite data streams by determining epoch lengths or restarting exhausted iterators.
This is a typed server-side library and payment gateway SDK for integrating Stripe into Node.js applications. It provides a typed client to manage payments, customers, and subscriptions, while offering specialized tools for executing secure financial transactions and managing billing resources. The library distinguishes itself through an idempotent API client that prevents duplicate operations using idempotency keys and exponential backoff retry logic. It includes a webhook signature validator to verify that incoming HTTPS event notifications are authentic and an async-iterator pagination wra
Uses JavaScript async iterators to stream paginated data from the API without buffering the entire payload.
Umbrella is a comprehensive ecosystem of TypeScript-based libraries and a mono-repository designed for UI rendering, mathematical frameworks, WebAssembly bridging, and functional data processing. It provides a suite of tools for managing reactive data streams, binary serialization, and specialized memory management. The project includes a reactive component model for generating HTML, SVG, and Canvas elements from nested data structures, as well as a system for integrating JavaScript and WebAssembly through generated bindings. It features a mathematical framework for linear algebra, tensor ope
Implements async iterators and reactive event streams for transforming continuous data flows in Deno.
xtensor is a C++ multidimensional array library for numerical computing that provides N-dimensional containers with an interface mirroring the NumPy API. It utilizes a lazy evaluation expression engine to defer numerical computations until assignment, which minimizes memory allocations and intermediate copies. The library features a foreign memory array adaptor that allows it to wrap external buffers, such as NumPy arrays, to perform numerical operations in-place without duplicating data. It further optimizes performance through lazy broadcasting and a system that manages the lifetime of temp
Provides memory-efficient, STL-compatible forward and reverse iterators to process tensor data.