3 repository-uri
Awesome GitHub RepositoriesLarge Dataset Processors
Utilities for chunking, sampling, and efficient file handling for datasets exceeding local memory capacity.
Distinct from Large-Scale Dataset Management: Distinct from Large-Scale Dataset Management: focuses on the processing and memory-efficient handling of large datasets within pipeline steps.
Explore 3 awesome GitHub repositories matching data & databases · Large Dataset Processors. Refine with filters or upvote what's useful.
Awesome Large Dataset Processors GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- zenml-io/zenml
zenml-io/zenml
5,451Vezi pe GitHubZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure. The platform distinguishes itself through a service-oriented
Implements data chunking, sampling, and efficient file formats to process datasets that exceed local memory capacity.
Pythonagentopsagentsai - danielbeach/data-engineering-practice
danielbeach/data-engineering-practice
2,726Vezi pe GitHubData engineering practice repository providing tutorials, distributed processing engines, and Python data pipeline automation scripts. The system encompasses automated data validation, distributed compute aggregation, embedded columnar querying, lazy evaluation planning, partitioned storage export, and cloud storage retrieval. The capability surface covers cloud integration and storage, data engineering and pipelines, data processing and analytics, data quality and testing, database and storage, file management, and monitoring and observability.
Processes datasets exceeding available system memory using streaming file lines.
Python - mahmoudparsian/data-algorithms-book
mahmoudparsian/data-algorithms-book
1,081Vezi pe GitHubThis repository is a collection of reference implementations and distributed data processing algorithms implemented in Java and Scala for cluster computing frameworks. It provides computational recipes for solving complex data processing problems, including large-scale dataset joins, aggregations, and word count tasks. The implementations cover both MapReduce paradigms and Apache Spark integrations, enabling programmatic job submission and execution across distributed node infrastructures. The collection includes specialized utilities for statistical analysis and text processing, such as data
Implements distributed algorithms for common analytical tasks like word counts, joins, and aggregations across large datasets.
Javaapache-hadoopapache-sparkdata-algorithms
