6 रिपॉजिटरी
Awesome GitHub RepositoriesBatch Data Processing
Capabilities for performing bulk operations on database records efficiently.
Distinguishing note: Specifically targets high-performance bulk insertion and modification of records, distinct from single-row CRUD.
Explore 6 awesome GitHub repositories matching data & databases · Batch Data Processing. Refine with filters or upvote what's useful.
Awesome Batch Data Processing GitHub Repositories
AI के साथ बेहतरीन रिपॉजिटरी खोजें।हम AI का उपयोग करके सबसे सटीक रिपॉजिटरी खोजेंगे।
- twentyhq/twenty
twentyhq/twenty
50,113GitHub पर देखेंTwenty is a headless customer relationship management framework that enables developers to build, version, and deploy custom business applications using code. By utilizing a declarative approach to data modeling, the platform allows for the definition of custom objects, fields, and complex relationships directly within the source code. This schema-driven architecture automatically generates corresponding REST and GraphQL APIs, ensuring that data structures and interface components remain synchronized across development and production environments. The platform distinguishes itself through a m
Enables efficient bulk creation, updates, and deletions of records through optimized API requests.
TypeScriptcrmcrm-systemcustomer - go-gorm/gorm
go-gorm/gorm
39,798GitHub पर देखेंGORM is a developer-focused object-relational mapping library for Go that provides a comprehensive data persistence framework. It serves as a database access layer, allowing developers to map application structures to database tables and perform CRUD operations using a fluent, type-safe query builder instead of writing raw SQL. The library distinguishes itself through its association-aware persistence, which automatically tracks and synchronizes complex entity relationships during database operations. It utilizes a driver-agnostic interface to maintain consistent behavior across various stora
To efficiently insert large number of records, pass a slice to the Create method. GORM will generate a single SQL statement to insert all the data and backfill primary key values, hook methods will be invoked too. It w
Gogogolanggorm - vonng/ddia
Vonng/ddia
22,648GitHub पर देखेंThis project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi
Processes large, fixed datasets as single units without modifying original input data.
Pythonbookdatabaseddia - unstructured-io/unstructured
Unstructured-IO/unstructured
14,019GitHub पर देखेंUnstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t
Automates the ingestion and transformation of large volumes of files from diverse storage sources.
HTMLdata-pipelinesdeep-learningdocument-image-analysis - rosedblabs/rosedb
rosedblabs/rosedb
4,878GitHub पर देखेंRoseDB एक पर्सिस्टेंट की-वैल्यू डेटाबेस और लॉग-स्ट्रक्चर्ड स्टोरेज इंजन है। यह एक हल्के स्टोरेज सिस्टम के रूप में कार्य करता है जो तेज़ डेटा रिट्रीवल और डिस्क-बैक्ड पर्सिस्टेंस प्रदान करने के लिए लॉग-स्ट्रक्चर्ड हैश टेबल और Bitcask इंजन इम्प्लीमेंटेशन का उपयोग करता है। यह सिस्टम एक एटॉमिक ट्रांजेक्शन इंजन के रूप में काम करता है, जो डेटा कंसिस्टेंसी बनाए रखने के लिए कई रीड और राइट ऑपरेशंस को सिंगल यूनिट्स में ग्रुप करता है। यह एक की-वैल्यू मॉडल के माध्यम से डेटा को हैंडल करता है जो व्यक्तिगत इंसर्शन, लुकअप और डिलीशन को सपोर्ट करता है। यह डेटाबेस बैच डेटा प्रोसेसिंग और एटॉमिक अपडेट्स के लिए क्षमताएं प्रदान करता है। इसके अतिरिक्त, प्रोजेक्ट में वर्ज़न्ड कंटेंट मैनेजमेंट और मल्टी-लैंग्वेज कंटेंट ट्रांसलेशन के फीचर्स भी शामिल हैं।
Optimizes performance by combining large volumes of read and write actions into single units.
Godata-structuresdatabaseembedded - bruin-data/ingestr
bruin-data/ingestr
3,714GitHub पर देखेंingestr is a command-line tool for copying and syncing data between different database engines and third-party platforms without writing custom code. It functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations. The tool features a schema-agnostic data loader that maps source fields to destination columns dynamically, removing the need for predefined static table definitions. It also operates as an incremental data synchronizer, updating destination tables by appending new records or merging changes to maintain current datasets. The system pr
Efficiently moves large volumes of data between systems using bulk operations and small chunks.
Go
