awesome-repositories.com
Blog
awesome-repositories.com

Découvrez les meilleurs dépôts open-source grâce à notre recherche par IA.

ExplorerRecherches sélectionnéesAlternatives open sourceLogiciels auto-hébergésBlogPlan du site
ProjetÀ proposNotre méthodologiePresseServeur MCP
Mentions légalesConfidentialitéConditions d'utilisation
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·

27 dépôts

Awesome GitHub RepositoriesItem Pipelines

Modular components that process individual data items as they move through an extraction or transformation pipeline.

Explore 27 awesome GitHub repositories matching data & databases · Item Pipelines. Refine with filters or upvote what's useful.

Awesome Item Pipelines GitHub Repositories

Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
  • safishamsi/graphifyAvatar de safishamsi

    safishamsi/graphify

    67,973Voir sur GitHub↗

    Graphify is a knowledge retrieval system that transforms directories of source code and documentation into structured, queryable project maps. It utilizes a code-to-graph parser to extract technical metadata and system connectivity, converting a mix of code, SQL schemas, and documentation into a unified graph structure. The project distinguishes itself by integrating these knowledge graphs with AI coding assistants through a Model Context Protocol server and dedicated tool hooks. This allows AI agents to perform lookups and impact analysis on node neighbors and shortest paths to understand ho

    Uses graph data to perform lookups on node neighbors and shortest paths to analyze how code changes affect the system.

    Pythonantigravityclaude-codecodex
    Voir sur GitHub↗67,973
  • egonex-ai/understand-anythingAvatar de Egonex-AI

    Egonex-AI/Understand-Anything

    66,456Voir sur GitHub↗

    Understand-Anything is a codebase architecture visualization tool that transforms source code and documentation into interactive knowledge graphs. It maps files, functions, and classes into a node-edge model to visualize architectural dependencies and project structures. The project provides specialized workflows for impact analysis, tracing connectivity paths from code modifications to identify affected downstream components. It also enables technical onboarding through automated architecture tours and the conversion of technical documentation into navigable networks of interconnected ideas.

    Traces connectivity paths from modified files to identify affected downstream architectural components.

    TypeScriptantigravity-skillsbusiness-knowledgeclaude-code
    Voir sur GitHub↗66,456
  • scrapy/scrapyAvatar de scrapy

    scrapy/scrapy

    62,274Voir sur GitHub↗

    Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

    Processes individual data items through a sequential chain of validation, cleaning, and storage handlers before persistence.

    Pythoncrawlercrawlingframework
    Voir sur GitHub↗62,274
  • facebook/rocksdbAvatar de facebook

    facebook/rocksdb

    31,767Voir sur GitHub↗

    RocksDB is a high-performance, embeddable persistent key-value library and storage engine based on Log-Structured Merge-trees. It is designed to provide durable storage for large-scale datasets, integrating directly into applications to manage data on flash and RAM-based hardware. The engine is distinguished by its focus on minimizing read and write amplification through multi-threaded compaction and custom memory allocators. It features specialized optimizations for flash storage, including support for zoned block devices, and provides the ability to extend store behavior via external plugin

    Automatically removes old data based on a configurable time-to-live (TTL) threshold.

    C++databasestorage-engine
    Voir sur GitHub↗31,767
  • fyne-io/fyneAvatar de fyne-io

    fyne-io/fyne

    27,941Voir sur GitHub↗

    Fyne is a cross-platform graphical user interface toolkit for the Go programming language. It provides a comprehensive framework for building native applications that run on desktop, mobile, and web environments from a single codebase. The toolkit centers on a canvas-based rendering engine and a device-independent layout engine, ensuring that visual elements maintain consistent dimensions and behavior across diverse operating systems and screen densities. The project distinguishes itself through a reactive data-binding system that automatically synchronizes application state with interface co

    Registers callback functions to automatically react to state changes in bound data items.

    Goandroidcross-platformfyne
    Voir sur GitHub↗27,941
  • pubkey/rxdbAvatar de pubkey

    pubkey/rxdb

    23,048Voir sur GitHub↗

    This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d

    Provides reactive streams for monitoring and responding to local document modifications in real time.

    TypeScriptangularbrowser-databasecouchdb
    Voir sur GitHub↗23,048
  • charmbracelet/gumAvatar de charmbracelet

    charmbracelet/gum

    22,814Voir sur GitHub↗

    Gum is a toolkit for building interactive, visually styled command-line interfaces and prompts directly within shell scripts. It functions as a library of modular components that allow developers to enhance terminal workflows by adding structured layouts, formatted text, and user-input widgets to standard command-line operations. The project distinguishes itself by providing a suite of specialized utilities for common shell tasks, such as fuzzy-matched selection menus, interactive file system navigation, and confirmation dialogs. It translates high-level styling and layout instructions into t

    Processes and displays text using templates to inject dynamic values into command-line output.

    Gobashshell
    Voir sur GitHub↗22,814
  • forem/foremAvatar de forem

    forem/forem

    22,726Voir sur GitHub↗

    Forem is an open-source platform designed for building and managing technical communities. It functions as a social publishing engine that enables members to share long-form content, participate in threaded discussions, and engage through social interactions. The platform provides tools for organizations to maintain branded profiles, host community hackathons, and facilitate collaborative learning through structured educational tracks. Beyond its social features, Forem integrates advanced capabilities for AI agent workflow orchestration and codebase knowledge graphing. It allows developers to

    Identifies which architectural components are affected by code modifications to assist in impact assessment.

    Rubycommunitydiscussionfeedback
    Voir sur GitHub↗22,726
  • prefecthq/prefectAvatar de PrefectHQ

    PrefectHQ/prefect

    21,640Voir sur GitHub↗

    Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

    Enables direct retrieval of individual database records by identifier and partition key.

    Pythonautomationdatadata-engineering
    Voir sur GitHub↗21,640
  • rust-lang/bookAvatar de rust-lang

    rust-lang/book

    17,930Voir sur GitHub↗

    The Rust Programming Language Book is the official technical guide and educational resource for the Rust language. It provides a comprehensive walkthrough of the language's design, focusing on its core identity as a systems programming language that enforces memory safety and high-performance execution without the need for a garbage collector. The project is distinguished by its focus on ownership, borrowing, and lifetime tracking, which allow the compiler to verify memory safety and thread safety at compile time. It covers the language's unique approach to zero-cost abstractions, including t

    Enables filtering of sequence elements using predicate closures for data processing.

    Rustbookmdbookrust
    Voir sur GitHub↗17,930
  • realm/realm-swiftAvatar de realm

    realm/realm-swift

    16,608Voir sur GitHub↗

    This is a mobile object database and NoSQL local data store that replaces relational tables with a schema-based model. It functions as a reactive data store, using live object observations and change notifications to trigger automatic user interface refreshes. The system provides built-in mobile cloud data synchronization to keep local datasets consistent with a remote server across multiple devices. It also includes security features for encrypted local storage, protecting sensitive on-disk data using at-rest encryption keys and fine-grained access control. Broad capabilities include object

    Provides mechanisms to observe data changes and trigger automatic user interface refreshes.

    Objective-C
    Voir sur GitHub↗16,608
  • mikefarah/yqAvatar de mikefarah

    mikefarah/yq

    14,913Voir sur GitHub↗

    This tool is a command-line processor designed for querying, updating, and transforming structured data files. It functions as a versatile engine for manipulating YAML, JSON, TOML, and XML documents, allowing users to perform complex operations directly from the terminal. By utilizing a path-based expression language, it enables precise navigation and modification of data structures within configuration files and infrastructure-as-code workflows. What distinguishes this tool is its ability to perform in-place document mutations while preserving original formatting, comments, and metadata. It

    The tool removes redundant elements from a collection while preserving the original sequence of the remaining items.

    Gobashclicsv
    Voir sur GitHub↗14,913
  • dexie/dexie.jsAvatar de dexie

    dexie/Dexie.js

    14,445Voir sur GitHub↗

    Dexie.js is a wrapper library for IndexedDB that provides a simplified interface for managing and querying structured data within the browser. It functions as a browser database manager used to maintain persistent application state and store binary blobs and records. The project serves as an offline-first data store that synchronizes browser data with remote servers to maintain consistency across sessions. It also acts as a reactive database store by monitoring data changes in real time to trigger automatic user interface updates, and functions as a client-side search engine for indexing and

    Offers mechanisms to register callbacks that trigger application updates whenever specific database subsets change.

    TypeScript
    Voir sur GitHub↗14,445
  • memcached/memcachedAvatar de memcached

    memcached/memcached

    14,132Voir sur GitHub↗

    Memcached is a high-performance, distributed, in-memory key-value storage and request routing engine. It functions as a volatile data store designed to accelerate dynamic applications by caching objects in RAM, thereby reducing backend database load and providing sub-millisecond response times. The system utilizes a specialized architecture that organizes memory into fixed-size slabs to minimize fragmentation and maximize throughput for high-concurrency workloads. The project distinguishes itself through a multi-threaded, lock-friendly design that scales across CPU cores and supports complex

    Removes cached items based on defined timestamps or background scanning to ensure data freshness.

    C
    Voir sur GitHub↗14,132
  • bytedance/flowgram.aiAvatar de bytedance

    bytedance/flowgram.ai

    8,146Voir sur GitHub↗

    Flowgram.ai is a workflow development framework for building AI workflow platforms. It provides a visual drag-and-drop canvas for constructing workflows, an Entity-Component-System (ECS) based document model for structuring workflow nodes as a tree, and a node-based form engine for managing configuration forms with built-in rendering, validation, side effects, and error handling. The framework also includes a workflow execution engine that parses directed graph workflows and runs nodes step by step with state tracking and array iteration. The framework distinguishes itself through a layered r

    Workflow builder watches updates to any entity data module on the canvas and triggers logic when changes occur.

    TypeScriptaiautomationcoze
    Voir sur GitHub↗8,146
  • dgraph-io/ristrettoAvatar de dgraph-io

    dgraph-io/ristretto

    6,932Voir sur GitHub↗

    Ristretto is a high-performance in-memory cache and concurrent key-value store for Go applications. It provides a thread-safe memory store that manages strict memory bounds and employs probabilistic set filters to reduce lookup overhead. The system is distinguished by an admission-policy cache that utilizes frequency sketches and cost-based eviction to maximize hit ratios. It minimizes contention and improves throughput through the use of striped ring buffers and concurrent map sharding. The project covers a broad range of data management capabilities, including time-based expiration, item f

    Automatically excludes items from retrieval if their predefined time-to-live expiration has passed.

    Go
    Voir sur GitHub↗6,932
  • ecrmnn/collect.jsAvatar de ecrmnn

    ecrmnn/collect.js

    6,571Voir sur GitHub↗

    collect.js is a dependency-free JavaScript library that provides a fluent, chainable interface for manipulating arrays and objects. It mirrors the Laravel Collection API, offering a consistent set of methods for data transformation across JavaScript and Laravel backend environments. The library stores collection data as plain arrays internally and supports fluent method chaining, where each method returns a new collection instance. The library distinguishes itself by closely replicating the Laravel Collection API in JavaScript, mapping each PHP method to an equivalent JavaScript implementatio

    Returns only items with specified keys or indices, or excludes items by key.

    JavaScriptcollectionlaravellaravel-collections
    Voir sur GitHub↗6,571
  • hazelcast/hazelcastAvatar de hazelcast

    hazelcast/hazelcast

    6,570Voir sur GitHub↗

    Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

    Applies transformation functions to each item in a stream to enrich or convert data structures.

    Javabig-datacachingdata-in-motion
    Voir sur GitHub↗6,570
  • rmax/scrapy-redisAvatar de rmax

    rmax/scrapy-redis

    5,639Voir sur GitHub↗

    Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica

    Pushes scraped data into Redis queues for downstream batch processing or consumption by separate services.

    Pythoncrawlerdistributedredis
    Voir sur GitHub↗5,639
  • qodo-ai/qodo-coverAvatar de qodo-ai

    qodo-ai/qodo-cover

    5,444Voir sur GitHub↗

    Qodo Cover est une plateforme de gouvernance d'ingénierie et un assistant piloté par l'IA conçu pour la revue de code automatisée et la génération de tests unitaires. Il utilise un graphe de connaissances de la base de code basé sur l'arbre de syntaxe abstraite (AST) pour mapper les dépendances et les relations architecturales, lui permettant d'analyser les pull requests et d'appliquer les standards de codage organisationnels. Le système se distingue par un pipeline d'analyse multi-agents qui effectue un raisonnement architectural et identifie des bugs au-delà du diff immédiat. Il dispose d'un serveur de protocole de contexte de modèle (MCP) pour exposer l'intelligence de la base de code à des outils externes et peut faire évoluer automatiquement les règles d'application en apprenant des décisions passées sur les pull requests. La plateforme fournit des capacités complètes pour la gestion des connaissances de la base de code, incluant l'exécution de recherches approfondies, le requêtage sémantique et le mapping des dépendances système. Elle inclut également des outils pour la génération itérative de tests unitaires afin d'augmenter la couverture de code et une remédiation automatisée pour appliquer des correctifs directement sur les pull requests. Les options de déploiement incluent le SaaS multi-tenant, le single-tenant ou des installations entièrement sur site (on-premises).

    Evaluates the operational impact of changes by analyzing cross-file relationships and historical patterns.

    Pythonagentsaitest-automation
    Voir sur GitHub↗5,444
Préc.12Suivant
  1. Home
  2. Data & Databases
  3. Data Processing Pipelines
  4. Processing Pipelines
  5. Item Pipelines

Explorer les sous-tags

  • Data Change Observers2 sous-tagsMechanisms for registering callbacks to receive notifications on data updates. **Distinct from Item Pipelines:** Focuses on reactive observation of bindable items rather than pipeline processing.
  • Data Item Filters4 sous-tagsLogic for excluding specific items from batch creation to handle invalid data. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on the exclusion/skipping logic rather than general transformation.
  • Database Item AccessorsInterfaces for retrieving individual records from database containers using unique identifiers. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on direct record retrieval rather than stream-based processing.
  • RedisPipelines that push scraped items into a Redis queue for asynchronous downstream processing. **Distinct from Item Pipelines:** Distinct from Item Pipelines: specifically pushes items into Redis queues for distributed consumption, not generic item processing.
  • Template RenderersComponents for defining reusable graphical structures for list entries. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on UI-level template rendering rather than data processing pipelines.