awesome-repositories.com
Blog
awesome-repositories.com

Descoperă cele mai bune repository-uri open source cu căutare AI.

ExploreazăCăutări recomandateAlternative open-sourceSoftware self-hostedBlogHartă site
ProiectDespreCum realizăm clasamentulPresăServer MCP
LegalConfidențialitateTermeni
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·

27 repository-uri

Awesome GitHub RepositoriesItem Pipelines

Modular components that process individual data items as they move through an extraction or transformation pipeline.

Explore 27 awesome GitHub repositories matching data & databases · Item Pipelines. Refine with filters or upvote what's useful.

Awesome Item Pipelines GitHub Repositories

Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
  • safishamsi/graphifyAvatar safishamsi

    safishamsi/graphify

    67,973Vezi pe GitHub↗

    Graphify is a knowledge retrieval system that transforms directories of source code and documentation into structured, queryable project maps. It utilizes a code-to-graph parser to extract technical metadata and system connectivity, converting a mix of code, SQL schemas, and documentation into a unified graph structure. The project distinguishes itself by integrating these knowledge graphs with AI coding assistants through a Model Context Protocol server and dedicated tool hooks. This allows AI agents to perform lookups and impact analysis on node neighbors and shortest paths to understand ho

    Uses graph data to perform lookups on node neighbors and shortest paths to analyze how code changes affect the system.

    Pythonantigravityclaude-codecodex
    Vezi pe GitHub↗67,973
  • egonex-ai/understand-anythingAvatar Egonex-AI

    Egonex-AI/Understand-Anything

    66,456Vezi pe GitHub↗

    Understand-Anything is a codebase architecture visualization tool that transforms source code and documentation into interactive knowledge graphs. It maps files, functions, and classes into a node-edge model to visualize architectural dependencies and project structures. The project provides specialized workflows for impact analysis, tracing connectivity paths from code modifications to identify affected downstream components. It also enables technical onboarding through automated architecture tours and the conversion of technical documentation into navigable networks of interconnected ideas.

    Traces connectivity paths from modified files to identify affected downstream architectural components.

    TypeScriptantigravity-skillsbusiness-knowledgeclaude-code
    Vezi pe GitHub↗66,456
  • scrapy/scrapyAvatar scrapy

    scrapy/scrapy

    62,274Vezi pe GitHub↗

    Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

    Processes individual data items through a sequential chain of validation, cleaning, and storage handlers before persistence.

    Pythoncrawlercrawlingframework
    Vezi pe GitHub↗62,274
  • facebook/rocksdbAvatar facebook

    facebook/rocksdb

    31,767Vezi pe GitHub↗

    RocksDB is a high-performance, embeddable persistent key-value library and storage engine based on Log-Structured Merge-trees. It is designed to provide durable storage for large-scale datasets, integrating directly into applications to manage data on flash and RAM-based hardware. The engine is distinguished by its focus on minimizing read and write amplification through multi-threaded compaction and custom memory allocators. It features specialized optimizations for flash storage, including support for zoned block devices, and provides the ability to extend store behavior via external plugin

    Automatically removes old data based on a configurable time-to-live (TTL) threshold.

    C++databasestorage-engine
    Vezi pe GitHub↗31,767
  • fyne-io/fyneAvatar fyne-io

    fyne-io/fyne

    27,941Vezi pe GitHub↗

    Fyne is a cross-platform graphical user interface toolkit for the Go programming language. It provides a comprehensive framework for building native applications that run on desktop, mobile, and web environments from a single codebase. The toolkit centers on a canvas-based rendering engine and a device-independent layout engine, ensuring that visual elements maintain consistent dimensions and behavior across diverse operating systems and screen densities. The project distinguishes itself through a reactive data-binding system that automatically synchronizes application state with interface co

    Registers callback functions to automatically react to state changes in bound data items.

    Goandroidcross-platformfyne
    Vezi pe GitHub↗27,941
  • pubkey/rxdbAvatar pubkey

    pubkey/rxdb

    23,048Vezi pe GitHub↗

    This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d

    Provides reactive streams for monitoring and responding to local document modifications in real time.

    TypeScriptangularbrowser-databasecouchdb
    Vezi pe GitHub↗23,048
  • charmbracelet/gumAvatar charmbracelet

    charmbracelet/gum

    22,814Vezi pe GitHub↗

    Gum is a toolkit for building interactive, visually styled command-line interfaces and prompts directly within shell scripts. It functions as a library of modular components that allow developers to enhance terminal workflows by adding structured layouts, formatted text, and user-input widgets to standard command-line operations. The project distinguishes itself by providing a suite of specialized utilities for common shell tasks, such as fuzzy-matched selection menus, interactive file system navigation, and confirmation dialogs. It translates high-level styling and layout instructions into t

    Processes and displays text using templates to inject dynamic values into command-line output.

    Gobashshell
    Vezi pe GitHub↗22,814
  • forem/foremAvatar forem

    forem/forem

    22,726Vezi pe GitHub↗

    Forem is an open-source platform designed for building and managing technical communities. It functions as a social publishing engine that enables members to share long-form content, participate in threaded discussions, and engage through social interactions. The platform provides tools for organizations to maintain branded profiles, host community hackathons, and facilitate collaborative learning through structured educational tracks. Beyond its social features, Forem integrates advanced capabilities for AI agent workflow orchestration and codebase knowledge graphing. It allows developers to

    Identifies which architectural components are affected by code modifications to assist in impact assessment.

    Rubycommunitydiscussionfeedback
    Vezi pe GitHub↗22,726
  • prefecthq/prefectAvatar PrefectHQ

    PrefectHQ/prefect

    21,640Vezi pe GitHub↗

    Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

    Enables direct retrieval of individual database records by identifier and partition key.

    Pythonautomationdatadata-engineering
    Vezi pe GitHub↗21,640
  • rust-lang/bookAvatar rust-lang

    rust-lang/book

    17,930Vezi pe GitHub↗

    The Rust Programming Language Book is the official technical guide and educational resource for the Rust language. It provides a comprehensive walkthrough of the language's design, focusing on its core identity as a systems programming language that enforces memory safety and high-performance execution without the need for a garbage collector. The project is distinguished by its focus on ownership, borrowing, and lifetime tracking, which allow the compiler to verify memory safety and thread safety at compile time. It covers the language's unique approach to zero-cost abstractions, including t

    Enables filtering of sequence elements using predicate closures for data processing.

    Rustbookmdbookrust
    Vezi pe GitHub↗17,930
  • realm/realm-swiftAvatar realm

    realm/realm-swift

    16,608Vezi pe GitHub↗

    This is a mobile object database and NoSQL local data store that replaces relational tables with a schema-based model. It functions as a reactive data store, using live object observations and change notifications to trigger automatic user interface refreshes. The system provides built-in mobile cloud data synchronization to keep local datasets consistent with a remote server across multiple devices. It also includes security features for encrypted local storage, protecting sensitive on-disk data using at-rest encryption keys and fine-grained access control. Broad capabilities include object

    Provides mechanisms to observe data changes and trigger automatic user interface refreshes.

    Objective-C
    Vezi pe GitHub↗16,608
  • mikefarah/yqAvatar mikefarah

    mikefarah/yq

    14,913Vezi pe GitHub↗

    This tool is a command-line processor designed for querying, updating, and transforming structured data files. It functions as a versatile engine for manipulating YAML, JSON, TOML, and XML documents, allowing users to perform complex operations directly from the terminal. By utilizing a path-based expression language, it enables precise navigation and modification of data structures within configuration files and infrastructure-as-code workflows. What distinguishes this tool is its ability to perform in-place document mutations while preserving original formatting, comments, and metadata. It

    The tool removes redundant elements from a collection while preserving the original sequence of the remaining items.

    Gobashclicsv
    Vezi pe GitHub↗14,913
  • dexie/dexie.jsAvatar dexie

    dexie/Dexie.js

    14,445Vezi pe GitHub↗

    Dexie.js is a wrapper library for IndexedDB that provides a simplified interface for managing and querying structured data within the browser. It functions as a browser database manager used to maintain persistent application state and store binary blobs and records. The project serves as an offline-first data store that synchronizes browser data with remote servers to maintain consistency across sessions. It also acts as a reactive database store by monitoring data changes in real time to trigger automatic user interface updates, and functions as a client-side search engine for indexing and

    Offers mechanisms to register callbacks that trigger application updates whenever specific database subsets change.

    TypeScript
    Vezi pe GitHub↗14,445
  • memcached/memcachedAvatar memcached

    memcached/memcached

    14,132Vezi pe GitHub↗

    Memcached is a high-performance, distributed, in-memory key-value storage and request routing engine. It functions as a volatile data store designed to accelerate dynamic applications by caching objects in RAM, thereby reducing backend database load and providing sub-millisecond response times. The system utilizes a specialized architecture that organizes memory into fixed-size slabs to minimize fragmentation and maximize throughput for high-concurrency workloads. The project distinguishes itself through a multi-threaded, lock-friendly design that scales across CPU cores and supports complex

    Removes cached items based on defined timestamps or background scanning to ensure data freshness.

    C
    Vezi pe GitHub↗14,132
  • bytedance/flowgram.aiAvatar bytedance

    bytedance/flowgram.ai

    8,146Vezi pe GitHub↗

    Flowgram.ai is a workflow development framework for building AI workflow platforms. It provides a visual drag-and-drop canvas for constructing workflows, an Entity-Component-System (ECS) based document model for structuring workflow nodes as a tree, and a node-based form engine for managing configuration forms with built-in rendering, validation, side effects, and error handling. The framework also includes a workflow execution engine that parses directed graph workflows and runs nodes step by step with state tracking and array iteration. The framework distinguishes itself through a layered r

    Workflow builder watches updates to any entity data module on the canvas and triggers logic when changes occur.

    TypeScriptaiautomationcoze
    Vezi pe GitHub↗8,146
  • dgraph-io/ristrettoAvatar dgraph-io

    dgraph-io/ristretto

    6,932Vezi pe GitHub↗

    Ristretto is a high-performance in-memory cache and concurrent key-value store for Go applications. It provides a thread-safe memory store that manages strict memory bounds and employs probabilistic set filters to reduce lookup overhead. The system is distinguished by an admission-policy cache that utilizes frequency sketches and cost-based eviction to maximize hit ratios. It minimizes contention and improves throughput through the use of striped ring buffers and concurrent map sharding. The project covers a broad range of data management capabilities, including time-based expiration, item f

    Automatically excludes items from retrieval if their predefined time-to-live expiration has passed.

    Go
    Vezi pe GitHub↗6,932
  • ecrmnn/collect.jsAvatar ecrmnn

    ecrmnn/collect.js

    6,571Vezi pe GitHub↗

    collect.js is a dependency-free JavaScript library that provides a fluent, chainable interface for manipulating arrays and objects. It mirrors the Laravel Collection API, offering a consistent set of methods for data transformation across JavaScript and Laravel backend environments. The library stores collection data as plain arrays internally and supports fluent method chaining, where each method returns a new collection instance. The library distinguishes itself by closely replicating the Laravel Collection API in JavaScript, mapping each PHP method to an equivalent JavaScript implementatio

    Returns only items with specified keys or indices, or excludes items by key.

    JavaScriptcollectionlaravellaravel-collections
    Vezi pe GitHub↗6,571
  • hazelcast/hazelcastAvatar hazelcast

    hazelcast/hazelcast

    6,570Vezi pe GitHub↗

    Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

    Applies transformation functions to each item in a stream to enrich or convert data structures.

    Javabig-datacachingdata-in-motion
    Vezi pe GitHub↗6,570
  • rmax/scrapy-redisAvatar rmax

    rmax/scrapy-redis

    5,639Vezi pe GitHub↗

    Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica

    Pushes scraped data into Redis queues for downstream batch processing or consumption by separate services.

    Pythoncrawlerdistributedredis
    Vezi pe GitHub↗5,639
  • qodo-ai/qodo-coverAvatar qodo-ai

    qodo-ai/qodo-cover

    5,444Vezi pe GitHub↗

    Qodo Cover este o platformă de guvernanță inginerească și un asistent bazat pe AI, conceput pentru revizuirea automată a codului și generarea de teste unitare. Utilizează un graf de cunoștințe al bazei de cod bazat pe arbore sintactic abstract (AST) pentru a mapa dependențele și relațiile arhitecturale, permițându-i să analizeze pull request-urile și să impună standardele organizaționale de codare. Sistemul se distinge printr-un pipeline de analiză multi-agent care efectuează raționamente arhitecturale și identifică bug-uri dincolo de diff-ul imediat. Dispune de un server de protocol de context al modelului pentru a expune inteligența bazei de cod către instrumente externe și poate evolua automat regulile de impunere învățând din deciziile istorice ale pull request-urilor. Platforma oferă capabilități cuprinzătoare pentru gestionarea cunoștințelor bazei de cod, inclusiv execuția de cercetări profunde, interogări semantice și maparea dependențelor sistemului. Include, de asemenea, instrumente pentru generarea iterativă de teste unitare pentru a crește acoperirea codului și remediere automată pentru a aplica corecții direct în pull request-uri. Opțiunile de deployment includ SaaS multi-tenant, single-tenant sau instalări complet on-premises.

    Evaluates the operational impact of changes by analyzing cross-file relationships and historical patterns.

    Pythonagentsaitest-automation
    Vezi pe GitHub↗5,444
Înapoi12Înainte
  1. Home
  2. Data & Databases
  3. Data Processing Pipelines
  4. Processing Pipelines
  5. Item Pipelines

Explorează sub-etichetele

  • Data Change Observers2 sub-tag-uriMechanisms for registering callbacks to receive notifications on data updates. **Distinct from Item Pipelines:** Focuses on reactive observation of bindable items rather than pipeline processing.
  • Data Item Filters4 sub-tag-uriLogic for excluding specific items from batch creation to handle invalid data. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on the exclusion/skipping logic rather than general transformation.
  • Database Item AccessorsInterfaces for retrieving individual records from database containers using unique identifiers. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on direct record retrieval rather than stream-based processing.
  • RedisPipelines that push scraped items into a Redis queue for asynchronous downstream processing. **Distinct from Item Pipelines:** Distinct from Item Pipelines: specifically pushes items into Redis queues for distributed consumption, not generic item processing.
  • Template RenderersComponents for defining reusable graphical structures for list entries. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on UI-level template rendering rather than data processing pipelines.