awesome-repositories.com
Blog
awesome-repositories.com

Entdecke die besten Open-Source-Repositories mit KI-gestützter Suche.

EntdeckenKuratierte SuchenOpen-Source-AlternativenSelf-hosted SoftwareBlogSitemap
ProjektÜber unsRanking-MethodikPresseMCP-Server
RechtlichesDatenschutzAGB
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·

27 Repos

Awesome GitHub RepositoriesItem Pipelines

Modular components that process individual data items as they move through an extraction or transformation pipeline.

Explore 27 awesome GitHub repositories matching data & databases · Item Pipelines. Refine with filters or upvote what's useful.

Awesome Item Pipelines GitHub Repositories

Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
  • safishamsi/graphifyAvatar von safishamsi

    safishamsi/graphify

    67,973Auf GitHub ansehen↗

    Graphify is a knowledge retrieval system that transforms directories of source code and documentation into structured, queryable project maps. It utilizes a code-to-graph parser to extract technical metadata and system connectivity, converting a mix of code, SQL schemas, and documentation into a unified graph structure. The project distinguishes itself by integrating these knowledge graphs with AI coding assistants through a Model Context Protocol server and dedicated tool hooks. This allows AI agents to perform lookups and impact analysis on node neighbors and shortest paths to understand ho

    Uses graph data to perform lookups on node neighbors and shortest paths to analyze how code changes affect the system.

    Pythonantigravityclaude-codecodex
    Auf GitHub ansehen↗67,973
  • egonex-ai/understand-anythingAvatar von Egonex-AI

    Egonex-AI/Understand-Anything

    66,456Auf GitHub ansehen↗

    Understand-Anything is a codebase architecture visualization tool that transforms source code and documentation into interactive knowledge graphs. It maps files, functions, and classes into a node-edge model to visualize architectural dependencies and project structures. The project provides specialized workflows for impact analysis, tracing connectivity paths from code modifications to identify affected downstream components. It also enables technical onboarding through automated architecture tours and the conversion of technical documentation into navigable networks of interconnected ideas.

    Traces connectivity paths from modified files to identify affected downstream architectural components.

    TypeScriptantigravity-skillsbusiness-knowledgeclaude-code
    Auf GitHub ansehen↗66,456
  • scrapy/scrapyAvatar von scrapy

    scrapy/scrapy

    62,274Auf GitHub ansehen↗

    Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

    Processes individual data items through a sequential chain of validation, cleaning, and storage handlers before persistence.

    Pythoncrawlercrawlingframework
    Auf GitHub ansehen↗62,274
  • facebook/rocksdbAvatar von facebook

    facebook/rocksdb

    31,767Auf GitHub ansehen↗

    RocksDB is a high-performance, embeddable persistent key-value library and storage engine based on Log-Structured Merge-trees. It is designed to provide durable storage for large-scale datasets, integrating directly into applications to manage data on flash and RAM-based hardware. The engine is distinguished by its focus on minimizing read and write amplification through multi-threaded compaction and custom memory allocators. It features specialized optimizations for flash storage, including support for zoned block devices, and provides the ability to extend store behavior via external plugin

    Automatically removes old data based on a configurable time-to-live (TTL) threshold.

    C++databasestorage-engine
    Auf GitHub ansehen↗31,767
  • fyne-io/fyneAvatar von fyne-io

    fyne-io/fyne

    27,941Auf GitHub ansehen↗

    Fyne is a cross-platform graphical user interface toolkit for the Go programming language. It provides a comprehensive framework for building native applications that run on desktop, mobile, and web environments from a single codebase. The toolkit centers on a canvas-based rendering engine and a device-independent layout engine, ensuring that visual elements maintain consistent dimensions and behavior across diverse operating systems and screen densities. The project distinguishes itself through a reactive data-binding system that automatically synchronizes application state with interface co

    Registers callback functions to automatically react to state changes in bound data items.

    Goandroidcross-platformfyne
    Auf GitHub ansehen↗27,941
  • pubkey/rxdbAvatar von pubkey

    pubkey/rxdb

    23,048Auf GitHub ansehen↗

    This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d

    Provides reactive streams for monitoring and responding to local document modifications in real time.

    TypeScriptangularbrowser-databasecouchdb
    Auf GitHub ansehen↗23,048
  • charmbracelet/gumAvatar von charmbracelet

    charmbracelet/gum

    22,814Auf GitHub ansehen↗

    Gum is a toolkit for building interactive, visually styled command-line interfaces and prompts directly within shell scripts. It functions as a library of modular components that allow developers to enhance terminal workflows by adding structured layouts, formatted text, and user-input widgets to standard command-line operations. The project distinguishes itself by providing a suite of specialized utilities for common shell tasks, such as fuzzy-matched selection menus, interactive file system navigation, and confirmation dialogs. It translates high-level styling and layout instructions into t

    Processes and displays text using templates to inject dynamic values into command-line output.

    Gobashshell
    Auf GitHub ansehen↗22,814
  • forem/foremAvatar von forem

    forem/forem

    22,726Auf GitHub ansehen↗

    Forem is an open-source platform designed for building and managing technical communities. It functions as a social publishing engine that enables members to share long-form content, participate in threaded discussions, and engage through social interactions. The platform provides tools for organizations to maintain branded profiles, host community hackathons, and facilitate collaborative learning through structured educational tracks. Beyond its social features, Forem integrates advanced capabilities for AI agent workflow orchestration and codebase knowledge graphing. It allows developers to

    Identifies which architectural components are affected by code modifications to assist in impact assessment.

    Rubycommunitydiscussionfeedback
    Auf GitHub ansehen↗22,726
  • prefecthq/prefectAvatar von PrefectHQ

    PrefectHQ/prefect

    21,640Auf GitHub ansehen↗

    Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

    Enables direct retrieval of individual database records by identifier and partition key.

    Pythonautomationdatadata-engineering
    Auf GitHub ansehen↗21,640
  • rust-lang/bookAvatar von rust-lang

    rust-lang/book

    17,930Auf GitHub ansehen↗

    The Rust Programming Language Book is the official technical guide and educational resource for the Rust language. It provides a comprehensive walkthrough of the language's design, focusing on its core identity as a systems programming language that enforces memory safety and high-performance execution without the need for a garbage collector. The project is distinguished by its focus on ownership, borrowing, and lifetime tracking, which allow the compiler to verify memory safety and thread safety at compile time. It covers the language's unique approach to zero-cost abstractions, including t

    Enables filtering of sequence elements using predicate closures for data processing.

    Rustbookmdbookrust
    Auf GitHub ansehen↗17,930
  • realm/realm-swiftAvatar von realm

    realm/realm-swift

    16,608Auf GitHub ansehen↗

    This is a mobile object database and NoSQL local data store that replaces relational tables with a schema-based model. It functions as a reactive data store, using live object observations and change notifications to trigger automatic user interface refreshes. The system provides built-in mobile cloud data synchronization to keep local datasets consistent with a remote server across multiple devices. It also includes security features for encrypted local storage, protecting sensitive on-disk data using at-rest encryption keys and fine-grained access control. Broad capabilities include object

    Provides mechanisms to observe data changes and trigger automatic user interface refreshes.

    Objective-C
    Auf GitHub ansehen↗16,608
  • mikefarah/yqAvatar von mikefarah

    mikefarah/yq

    14,913Auf GitHub ansehen↗

    This tool is a command-line processor designed for querying, updating, and transforming structured data files. It functions as a versatile engine for manipulating YAML, JSON, TOML, and XML documents, allowing users to perform complex operations directly from the terminal. By utilizing a path-based expression language, it enables precise navigation and modification of data structures within configuration files and infrastructure-as-code workflows. What distinguishes this tool is its ability to perform in-place document mutations while preserving original formatting, comments, and metadata. It

    The tool removes redundant elements from a collection while preserving the original sequence of the remaining items.

    Gobashclicsv
    Auf GitHub ansehen↗14,913
  • dexie/dexie.jsAvatar von dexie

    dexie/Dexie.js

    14,445Auf GitHub ansehen↗

    Dexie.js is a wrapper library for IndexedDB that provides a simplified interface for managing and querying structured data within the browser. It functions as a browser database manager used to maintain persistent application state and store binary blobs and records. The project serves as an offline-first data store that synchronizes browser data with remote servers to maintain consistency across sessions. It also acts as a reactive database store by monitoring data changes in real time to trigger automatic user interface updates, and functions as a client-side search engine for indexing and

    Offers mechanisms to register callbacks that trigger application updates whenever specific database subsets change.

    TypeScript
    Auf GitHub ansehen↗14,445
  • memcached/memcachedAvatar von memcached

    memcached/memcached

    14,132Auf GitHub ansehen↗

    Memcached is a high-performance, distributed, in-memory key-value storage and request routing engine. It functions as a volatile data store designed to accelerate dynamic applications by caching objects in RAM, thereby reducing backend database load and providing sub-millisecond response times. The system utilizes a specialized architecture that organizes memory into fixed-size slabs to minimize fragmentation and maximize throughput for high-concurrency workloads. The project distinguishes itself through a multi-threaded, lock-friendly design that scales across CPU cores and supports complex

    Removes cached items based on defined timestamps or background scanning to ensure data freshness.

    C
    Auf GitHub ansehen↗14,132
  • bytedance/flowgram.aiAvatar von bytedance

    bytedance/flowgram.ai

    8,146Auf GitHub ansehen↗

    Flowgram.ai is a workflow development framework for building AI workflow platforms. It provides a visual drag-and-drop canvas for constructing workflows, an Entity-Component-System (ECS) based document model for structuring workflow nodes as a tree, and a node-based form engine for managing configuration forms with built-in rendering, validation, side effects, and error handling. The framework also includes a workflow execution engine that parses directed graph workflows and runs nodes step by step with state tracking and array iteration. The framework distinguishes itself through a layered r

    Workflow builder watches updates to any entity data module on the canvas and triggers logic when changes occur.

    TypeScriptaiautomationcoze
    Auf GitHub ansehen↗8,146
  • dgraph-io/ristrettoAvatar von dgraph-io

    dgraph-io/ristretto

    6,932Auf GitHub ansehen↗

    Ristretto is a high-performance in-memory cache and concurrent key-value store for Go applications. It provides a thread-safe memory store that manages strict memory bounds and employs probabilistic set filters to reduce lookup overhead. The system is distinguished by an admission-policy cache that utilizes frequency sketches and cost-based eviction to maximize hit ratios. It minimizes contention and improves throughput through the use of striped ring buffers and concurrent map sharding. The project covers a broad range of data management capabilities, including time-based expiration, item f

    Automatically excludes items from retrieval if their predefined time-to-live expiration has passed.

    Go
    Auf GitHub ansehen↗6,932
  • ecrmnn/collect.jsAvatar von ecrmnn

    ecrmnn/collect.js

    6,571Auf GitHub ansehen↗

    collect.js is a dependency-free JavaScript library that provides a fluent, chainable interface for manipulating arrays and objects. It mirrors the Laravel Collection API, offering a consistent set of methods for data transformation across JavaScript and Laravel backend environments. The library stores collection data as plain arrays internally and supports fluent method chaining, where each method returns a new collection instance. The library distinguishes itself by closely replicating the Laravel Collection API in JavaScript, mapping each PHP method to an equivalent JavaScript implementatio

    Returns only items with specified keys or indices, or excludes items by key.

    JavaScriptcollectionlaravellaravel-collections
    Auf GitHub ansehen↗6,571
  • hazelcast/hazelcastAvatar von hazelcast

    hazelcast/hazelcast

    6,570Auf GitHub ansehen↗

    Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

    Applies transformation functions to each item in a stream to enrich or convert data structures.

    Javabig-datacachingdata-in-motion
    Auf GitHub ansehen↗6,570
  • rmax/scrapy-redisAvatar von rmax

    rmax/scrapy-redis

    5,639Auf GitHub ansehen↗

    Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica

    Pushes scraped data into Redis queues for downstream batch processing or consumption by separate services.

    Pythoncrawlerdistributedredis
    Auf GitHub ansehen↗5,639
  • qodo-ai/qodo-coverAvatar von qodo-ai

    qodo-ai/qodo-cover

    5,444Auf GitHub ansehen↗

    Qodo Cover ist eine Engineering-Governance-Plattform und ein KI-gestützter Assistent für automatisierte Code-Reviews und Unit-Test-Generierung. Es nutzt einen Wissensgraphen auf Basis des abstrakten Syntaxbaums (AST), um Abhängigkeiten und architektonische Beziehungen abzubilden, wodurch Pull Requests analysiert und organisatorische Coding-Standards durchgesetzt werden können. Das System zeichnet sich durch eine Multi-Agenten-Analyse-Pipeline aus, die architektonische Schlussfolgerungen zieht und Fehler identifiziert, die über das unmittelbare Diff hinausgehen. Es verfügt über einen Model-Context-Protocol-Server, um Codebase-Intelligenz für externe Tools verfügbar zu machen, und kann Durchsetzungsregeln automatisch weiterentwickeln, indem es aus historischen Pull-Request-Entscheidungen lernt. Die Plattform bietet umfassende Funktionen für das Wissensmanagement der Codebase, einschließlich Deep-Research-Ausführung, semantischer Abfragen und System-Abhängigkeits-Mapping. Sie enthält zudem Werkzeuge zur iterativen Unit-Test-Generierung zur Erhöhung der Code-Abdeckung sowie automatisierte Remediation zur direkten Anwendung von Fixes auf Pull Requests. Bereitstellungsoptionen umfassen Multi-Tenant-SaaS, Single-Tenant oder vollständig On-Premises-Installationen.

    Evaluates the operational impact of changes by analyzing cross-file relationships and historical patterns.

    Pythonagentsaitest-automation
    Auf GitHub ansehen↗5,444
Vorherige12Nächste
  1. Home
  2. Data & Databases
  3. Data Processing Pipelines
  4. Processing Pipelines
  5. Item Pipelines

Unter-Tags erkunden

  • Data Change Observers2 Sub-TagsMechanisms for registering callbacks to receive notifications on data updates. **Distinct from Item Pipelines:** Focuses on reactive observation of bindable items rather than pipeline processing.
  • Data Item Filters4 Sub-TagsLogic for excluding specific items from batch creation to handle invalid data. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on the exclusion/skipping logic rather than general transformation.
  • Database Item AccessorsInterfaces for retrieving individual records from database containers using unique identifiers. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on direct record retrieval rather than stream-based processing.
  • RedisPipelines that push scraped items into a Redis queue for asynchronous downstream processing. **Distinct from Item Pipelines:** Distinct from Item Pipelines: specifically pushes items into Redis queues for distributed consumption, not generic item processing.
  • Template RenderersComponents for defining reusable graphical structures for list entries. **Distinct from Item Pipelines:** Distinct from Item Pipelines: focuses on UI-level template rendering rather than data processing pipelines.