awesome-repositories.com
Blog
awesome-repositories.com

Descubre los mejores repositorios open-source con nuestra búsqueda potenciada por IA.

ExplorarBúsquedas curadasAlternativas open-sourceSoftware autohospedableBlogMapa del sitio
ProyectoAcerca deCómo clasificamosPrensaServidor MCP
Aviso legalPrivacidadTérminos
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·

218 repositorios

Awesome GitHub RepositoriesData Handling and Throughput

Methods for optimizing the ingestion, transfer, and memory management of large datasets.

Explore 218 awesome GitHub repositories matching software engineering & architecture · Data Handling and Throughput. Refine with filters or upvote what's useful.

Awesome Data Handling and Throughput GitHub Repositories

Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
  • kamranahmedse/developer-roadmapAvatar de kamranahmedse

    kamranahmedse/developer-roadmap

    357,434Ver en GitHub↗

    Developer Roadmap es una plataforma impulsada por la comunidad que proporciona rutas de aprendizaje estructuradas basadas en grafos para la ingeniería de software. Sirve como un repositorio de conocimiento integral donde los dominios técnicos se organizan en secuencias visuales para guiar la adquisición de habilidades profesionales y el crecimiento profesional. El proyecto se distingue por un ecosistema colaborativo que permite a los usuarios contribuir con roadmaps, curar las mejores prácticas de la industria y mantener perfiles profesionales. Integra marcos de evaluación de diagnóstico para evaluar la competencia técnica, ayudando a los desarrolladores a identificar brechas de conocimiento y prepararse para entrevistas profesionales a través de secuencias de aprendizaje específicas. Más allá de sus capacidades principales de mapeo, la plataforma ofrece ideas de proyectos prácticos y tutoría interactiva para reforzar los conceptos de ingeniería. Proporciona un espacio centralizado para que la comunidad comparta recursos, rastree el desarrollo progresivo de habilidades y navegue por paisajes técnicos complejos.

    Manages object duplication for roadmap state and configuration.

    TypeScriptangular-roadmapbackend-roadmapblockchain-roadmap
    Ver en GitHub↗357,434
  • facebook/reactAvatar de facebook

    facebook/react

    245,669Ver en GitHub↗

    React es una biblioteca de JavaScript para crear interfaces de usuario basadas en una arquitectura orientada a componentes y un flujo de datos unidireccional.

    Updates object state by creating new copies with overridden fields using spread syntax.

    JavaScriptjavascriptuifrontend
    Ver en GitHub↗245,669
  • vuejs/vueAvatar de vuejs

    vuejs/vue

    209,900Ver en GitHub↗

    Vue es un framework de JavaScript progresivo basado en componentes diseñado para construir interfaces de usuario reactivas y aplicaciones de una sola página. Se centra en un sistema de plantillas declarativo que transforma HTML en funciones de renderizado eficientes, permitiendo a los desarrolladores organizar interfaces complejas en unidades aisladas y reutilizables que se sincronizan automáticamente con el estado de la aplicación. El framework se distingue por un sistema de reactividad de seguimiento de dependencias que monitorea el acceso a los datos durante el renderizado para activar actualizaciones precisas. Proporciona una arquitectura flexible que admite tanto la adopción incremental como una biblioteca ligera como el desarrollo de aplicaciones a gran escala. Los desarrolladores pueden aprovechar un modelo de extensibilidad basado en plugins robusto para inyectar lógica global, mientras que la reconciliación del DOM virtual del framework asegura actualizaciones de interfaz eficientes calculando mutaciones mínimas. Más allá de sus capacidades de renderizado principales, el proyecto incluye un conjunto completo de herramientas para gestionar el estado de la aplicación, enrutamiento basado en URL y renderizado del lado del servidor. Ofrece un amplio soporte para la composición de componentes, distribución de contenido y gestión de animaciones, junto con medidas de seguridad integradas como el escape automático de contenido para prevenir vulnerabilidades comunes. El framework se distribuye con declaraciones de tipo oficiales para admitir el análisis estático y puede instalarse a través de gestores de paquetes estándar o integrarse directamente en entornos de navegador a través de etiquetas de script.

    Optimizes performance by rendering static content only once and skipping future updates.

    TypeScriptframeworkfrontendjavascript
    Ver en GitHub↗209,900
  • yt-dlp/yt-dlpAvatar de yt-dlp

    yt-dlp/yt-dlp

    170,963Ver en GitHub↗

    This project is a command-line media downloader designed for the systematic retrieval and organization of digital content from diverse online platforms. It functions as an extensible extraction engine that utilizes a declarative format-selection pipeline to automate the identification, merging, and downloading of specific audio and video streams based on user-defined criteria. The system distinguishes itself through a modular architecture that supports custom plugins and site-specific scripts, allowing for the bypass of platform restrictions and the handling of complex authentication challeng

    Controls concurrent fragment downloads, rate limits, and automated retry policies to optimize resource usage during network operations.

    Pythonclidownloaderpython
    Ver en GitHub↗170,963
  • chalarangelo/30-seconds-of-codeAvatar de Chalarangelo

    Chalarangelo/30-seconds-of-code

    128,121Ver en GitHub↗

    30-seconds-of-code is a comprehensive knowledge base and programming snippet library designed to support software engineering education and professional development. It provides a curated collection of reusable code units and technical guides that help developers master core language mechanics, design patterns, and architectural philosophies. The project distinguishes itself by offering a wide-ranging library of algorithmic solutions and web development patterns that are organized into modular, independently testable units. It emphasizes functional programming paradigms and declarative logic,

    Provides utility functions for creating deep, independent copies of complex data structures.

    JavaScriptastroawesome-listcss
    Ver en GitHub↗128,121
  • syncthing/syncthingAvatar de syncthing

    syncthing/syncthing

    85,400Ver en GitHub↗

    Syncthing es un motor de sincronización de archivos descentralizado que mantiene estados de datos consistentes en múltiples dispositivos a través de redes mesh peer-to-peer. Funciona como un demonio en segundo plano que replica automáticamente las creaciones, modificaciones y eliminaciones de archivos entre nodos de confianza sin necesidad de servidores centrales. Al utilizar indexación de bloques direccionable por contenido y sincronización delta a nivel de bloque, el sistema identifica y transfiere solo los segmentos modificados de los archivos, asegurando una propagación de datos eficiente en entornos heterogéneos. El proyecto se distingue por una arquitectura centrada en la seguridad que se basa en la autenticación TLS mutua para verificar la identidad del dispositivo, asegurando que todas las conexiones estén vinculadas criptográficamente a huellas digitales de certificados de confianza. Admite modos de sincronización flexibles, incluyendo replicación bidireccional, duplicación unidireccional para copias de seguridad y cumplimiento basado en referencias. Para mayor privacidad, el sistema proporciona cifrado a nivel de carpeta para dispositivos no confiables y permite un control granular del tráfico de red, incluida la capacidad de restringir operaciones a redes locales o utilizar infraestructura de retransmisión para el cruce de NAT. Más allá de sus capacidades principales de replicación, la plataforma ofrece herramientas de gestión integrales, incluido un panel basado en web para monitorear el estado de la conexión y el rendimiento, así como una interfaz de línea de comandos para una configuración avanzada. Incluye estrategias de versionado robustas para proteger contra la pérdida de datos y admite escenarios de despliegue complejos a través de la integración de servicios nativos y métricas de observabilidad. El software está diseñado para la compatibilidad multiplataforma y puede instalarse a través de gestores de paquetes estándar o entornos contenedorizados.

    Optimizes network performance by adjusting concurrency limits and resource usage based on environmental capacity.

    Gogop2ppeer-to-peer
    Ver en GitHub↗85,400
  • infiniflow/ragflowAvatar de infiniflow

    infiniflow/ragflow

    82,922Ver en GitHub↗

    This project is a comprehensive retrieval-augmented generation platform designed for building, managing, and deploying knowledge-based AI applications. It provides a unified environment for organizing datasets, configuring conversational chat assistants, and developing autonomous agents that execute multi-step reasoning workflows. By integrating document intelligence with advanced retrieval pipelines, the platform enables the creation of grounded, verifiable responses supported by traceable citations. The platform distinguishes itself through deep document understanding and sophisticated know

    Tunes batch processing, OCR engines, and parsing services to minimize latency in retrieval-augmented generation pipelines.

    Pythonagentagenticagentic-ai
    Ver en GitHub↗82,922
  • base/nodeAvatar de base

    base/node

    68,511Ver en GitHub↗

    This project is a containerized implementation of a blockchain node for the Base network. It manages the execution and consensus processes of an Ethereum Virtual Machine node within an isolated environment to maintain the network and process blockchain data. The infrastructure includes a specialized interface for processing pending blocks via websockets to reduce latency. It also provides a state synchronization tool that uses pre-computed snapshots to accelerate the process of reaching the current block height. The system covers network identity configuration for mainnet and testnet environ

    Offers controls for data pruning, logging levels, and metric exports to optimize node performance and storage.

    Shell
    Ver en GitHub↗68,511
  • chartjs/chart.jsAvatar de chartjs

    chartjs/Chart.js

    67,526Ver en GitHub↗

    Chart.js is a declarative data visualization framework that renders interactive, responsive charts directly onto an HTML5 canvas element. It functions as a configuration-driven engine, transforming structured datasets into complex graphical representations by merging user-defined settings with global defaults. The library utilizes a high-performance rendering pipeline that executes drawing commands during each animation frame to maintain smooth visual feedback. The project distinguishes itself through a modular, extensible architecture that allows developers to register custom scales, control

    Filters or samples large datasets to reduce computational overhead before passing them to the rendering engine.

    JavaScriptcanvaschartgraph
    Ver en GitHub↗67,526
  • ffmpeg/ffmpegAvatar de FFmpeg

    FFmpeg/FFmpeg

    61,176Ver en GitHub↗

    FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f

    Transfers audio and video packets between containers without decoding or re-encoding to ensure high-speed processing with zero quality loss.

    Caudiocffmpeg
    Ver en GitHub↗61,176
  • addyosmani/agent-skillsAvatar de addyosmani

    addyosmani/agent-skills

    60,849Ver en GitHub↗

    Agent-skills is a collection of structured instructions and behavioral personas designed to standardize how AI coding agents perform engineering tasks. It functions as a workflow orchestrator that maps natural language intent to repeatable technical sequences and verification checklists. The project distinguishes itself through the use of specialized markdown-defined roles, such as security auditors or test engineers, to apply targeted domain expertise. It employs an evidence-based verification model that requires runtime data or passing tests as mandatory exit criteria to ensure AI-generated

    Applies a measure-first approach and bundle analysis to identify performance regressions and improve stability.

    Shellagent-skillsantigravityantigravity-ide
    Ver en GitHub↗60,849
  • xingshaocheng/architect-awesomeAvatar de xingshaocheng

    xingshaocheng/architect-awesome

    60,821Ver en GitHub↗

    This project serves as a comprehensive knowledge base and reference for distributed systems engineering and enterprise software architecture. It provides a structured collection of technical resources, design patterns, and methodologies intended to assist in the design, maintenance, and scaling of complex, high-performance software environments. The repository distinguishes itself by offering deep dives into core architectural concepts such as actor-based concurrency, aspect-oriented interception, and inversion-of-control containers. It emphasizes the practical application of distributed syst

    Optimize high-throughput operations by transferring data directly between memory buffers to reduce CPU cycles.

    Ver en GitHub↗60,821
  • vuejs/coreAvatar de vuejs

    vuejs/core

    53,861Ver en GitHub↗

    Vue is a progressive JavaScript framework designed for building modular, reactive user interfaces. It utilizes a component-based architecture that allows developers to encapsulate logic, templates, and styles into reusable units. At its core, the framework employs a virtual DOM renderer and a proxy-based reactivity system to synchronize application state with the document object model efficiently. What distinguishes this framework is its focus on developer experience and flexibility. It supports a single-file component format that colocalizes related concerns, alongside a powerful composition

    Optimizes rendering performance for massive datasets by employing list virtualization and bypassing deep reactivity for immutable data structures.

    TypeScript
    Ver en GitHub↗53,861
  • etcd-io/etcdAvatar de etcd-io

    etcd-io/etcd

    51,838Ver en GitHub↗

    etcd is a distributed, strongly consistent key-value store designed to provide reliable storage for critical system metadata and coordination primitives. It functions as a distributed consensus engine, utilizing a replicated log and leader-based state machine to ensure that all nodes in a cluster maintain a synchronized view of data. By providing atomic operations and linearizable reads and writes, it serves as a foundational component for distributed systems requiring high availability and fault tolerance. The system distinguishes itself through its multi-version concurrency control, which e

    Analysis of latency and throughput metrics identifies performance bottlenecks to optimize storage engine behavior for high-demand applications.

    Gocncfconsensusdatabase
    Ver en GitHub↗51,838
  • cwida/duckdbAvatar de cwida

    cwida/duckdb

    38,822Ver en GitHub↗

    DuckDB es una base de datos SQL analítica embebida y en proceso, y un sistema de gestión de bases de datos OLAP. Funciona como un motor de datos para archivos Parquet y CSV, permitiendo a los usuarios ejecutar consultas SQL complejas en grandes conjuntos de datos sin necesidad de un proceso de servidor independiente. El sistema está diseñado para el procesamiento analítico local y flujos de trabajo de ciencia de datos embebidos. Permite la consulta y el análisis directo de archivos Parquet y CSV desde el disco, evitando la necesidad de cargar datos en una base de datos permanente. El motor proporciona una ejecución SQL analítica de alto rendimiento, incluyendo soporte para funciones de ventana y subconsultas anidadas. Incorpora un diseño de almacenamiento en columnas y ejecución de consultas vectorizadas para manejar la manipulación y exploración de datos a gran escala. La base de datos es accesible a través de una interfaz de línea de comandos independiente y enlaces específicos de lenguaje para Python, R, Java y Wasm.

    Optimizes the processing of massive structured datasets using high-performance SQL queries and window functions.

    C++
    Ver en GitHub↗38,822
  • aobingjava/javafamilyAvatar de AobingJava

    AobingJava/JavaFamily

    36,959Ver en GitHub↗

    JavaFamily is a curated set of learning paths and reference guides for backend engineering, distributed systems, and virtual machine internals. It provides a structured curriculum covering the Java language, operating system concepts, and network protocols. The project features detailed study guides for the Java virtual machine architecture, including memory management and garbage collection. It also includes a comprehensive reference for distributed systems, covering microservices, remote procedure call frameworks, and scalable system design. The collection covers a broad range of technical

    Explains zero-copy techniques for transferring data between buffers to increase I/O processing speeds.

    interviewjavajava8
    Ver en GitHub↗36,959
  • facebook/immutable-jsAvatar de facebook

    facebook/immutable-js

    33,060Ver en GitHub↗

    This is a persistent data structure library for JavaScript that provides collections which prevent the direct mutation of objects and arrays. It serves as an immutable state management tool and functional programming utility, ensuring that data remains unchanged after creation to simplify change detection and state tracking. The library enables the maintenance of application state by producing new versions of data structures during updates. It focuses on efficient data comparison by checking actual content instead of memory references and supports a functional programming workflow to prevent

    Optimizes the processing of large datasets through lazy sequence evaluation and batched mutations.

    TypeScript
    Ver en GitHub↗33,060
  • sillytavern/sillytavernAvatar de SillyTavern

    SillyTavern/SillyTavern

    29,463Ver en GitHub↗

    SillyTavern is a comprehensive interface and orchestration platform designed for immersive AI roleplay and interactive chat experiences. It functions as a unified gateway that connects users to a wide array of local and cloud-based large language models, providing a centralized environment to manage complex character personas, narrative context, and model-driven interactions. The platform distinguishes itself through its advanced prompt engineering and automation capabilities. It utilizes a sophisticated macro-based templating engine and vector-database retrieval to dynamically inject lore, c

    Improves responsiveness by caching character data, compressing payloads, and generating media thumbnails.

    JavaScriptaichatllm
    Ver en GitHub↗29,463
  • projectdiscovery/nucleiAvatar de projectdiscovery

    projectdiscovery/nuclei

    29,189Ver en GitHub↗

    Nuclei is a modular security scanning framework designed for automated vulnerability detection and infrastructure reconnaissance. It functions as a template-driven engine that executes security checks across diverse network protocols, allowing users to define custom detection logic to identify vulnerabilities, misconfigurations, and exposed assets. The platform distinguishes itself through its highly extensible architecture, which supports distributed scanning, headless browser automation for dynamic web content, and out-of-band interaction monitoring to detect blind vulnerabilities. It integ

    Controls the speed and intensity of requests using concurrency, parallelism, and rate-limiting settings to avoid triggering security blocks.

    Goattack-surfacecve-scannerdast
    Ver en GitHub↗29,189
  • ajaxorg/aceAvatar de ajaxorg

    ajaxorg/ace

    27,133Ver en GitHub↗

    Ace is a high-performance, browser-based code editor and JavaScript text editor component designed to be embedded into web applications. It functions as a feature-rich alternative to standard text areas, providing a syntax highlighting engine and support for multiple programming and markup languages. The editor is distinguished by its ability to handle large documents containing millions of lines while maintaining stable performance. It supports multi-cursor editing, allowing simultaneous modifications at multiple selection points across a document. The project covers a broad range of develo

    Optimizes browser performance to allow stable editing of text files containing millions of lines.

    JavaScript
    Ver en GitHub↗27,133
Ant.123456…11Siguiente
  1. Home
  2. Software Engineering & Architecture
  3. Performance and Reliability
  4. Performance Optimization
  5. Data Handling and Throughput

Explorar subetiquetas

  • Data Decimation StrategiesTechniques that reduce computational load by filtering or sampling large datasets before rendering or processing.
  • Download Throughput ControlsControls that manage data transfer rates, concurrency limits, and resource usage during network download operations.
  • Large Dataset Optimizations22 sub-etiquetasOptimization strategies that improve performance when processing or rendering large volumes of data.
  • RAG Pipeline OptimizersConfigurations and tools for tuning document parsing and retrieval latency in RAG systems.
  • System Performance Optimization4 sub-etiquetasPractices for analyzing system metrics to identify and resolve performance bottlenecks in throughput and latency.
  • Zero-Copy Mechanisms3 sub-etiquetasTechniques for transferring data between memory buffers without redundant copying to minimize CPU overhead.