Awesome GitHub RepositoriesLarge File Processing

EasyExcel is a Java processing library designed for reading and writing XLS, XLSX, and CSV files. It functions as a memory-efficient spreadsheet parser, an object-relational mapper that binds spreadsheet columns to Java class fields, and a stream-based exporter for handling high-volume data. The library distinguishes itself through a streaming model that processes large files row-by-row via listeners to prevent heap memory overflow. It also operates as a template engine, allowing the population of predefined spreadsheet files with dynamic data while preserving original layouts and styles. Br

ExcelJS is a Node.js spreadsheet engine and manipulation library used for reading, writing, and modifying XLSX and CSV files. It functions as a formatting tool and asynchronous streaming parser for generating complex workbooks containing formulas, rich text, and custom styles. The library is distinguished by its ability to process large datasets using asynchronous data streaming and incremental processing, which minimizes memory usage during data extraction and file generation. Its capability surface covers comprehensive data management, including structured tables, named ranges, and cell da

PHPExcel is a PHP spreadsheet library used for programmatically reading and writing spreadsheet files in various formats. It utilizes an in-memory spreadsheet model that maps spreadsheet structures to a hierarchy of objects for programmatic manipulation. The library functions as an Office Open XML processor for generating and manipulating XLSX documents and serves as a reader for extracting data and structure from legacy binary XLS files. It also includes tools for CSV data integration and importing. The project provides capabilities for automated report generation and spreadsheet data extra

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

The 1BRC (One Billion Row Challenge) is a Java performance benchmarking exercise that processes one billion temperature records from a text file to compute the minimum, mean, and maximum temperature per weather station. At its core, it is a large-scale data aggregation challenge designed to test how efficiently a Java program can parse and aggregate structured data from a plain text file, serving as both a programming exercise and a benchmark for Java performance optimization. The project distinguishes itself through a collection of performance-oriented architectural patterns for high-through

Dawarich is a self-hosted location history manager and travel journaling platform. It functions as a personal travel archive that collects GPS coordinates and movement data, providing a private alternative to proprietary tracking services. The system utilizes a PostgreSQL geospatial database to store coordinates, visits, and custom geofence boundaries. The project distinguishes itself as a geospatial data converter and visualization tool, capable of transforming location history between formats such as GPX, KML, and GeoJSON. It allows users to organize GPS tracks and geotagged photos into nam

Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management. The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer

Apache Fesod is a lightweight Java library that wraps Apache POI to provide a streaming API for reading and writing large Excel files. Its core identity is a low-memory spreadsheet processor that prevents out-of-memory errors by handling data row by row, never loading an entire document into memory at once. The library distinguishes itself through a listener-driven event model that fires row-level events to user code as each row is parsed, enabling incremental processing. It also includes an object mapping layer that maps spreadsheet rows directly to Java objects using configurable column map

Cocoindex is an incremental data processing engine that builds and maintains live indexes for AI agents, with a core focus on codebase indexing and knowledge graph extraction. The engine uses a function-graph execution model where user-defined Python functions are composed into a directed acyclic graph, and it processes data incrementally so only changed source records or code paths are re-computed, avoiding full recomputation at any scale. It supports automatic schema inference from transformation pipeline type annotations and provides full data lineage tracing, tagging every output record wi

Zed is a terminal-based code editor built in Rust that provides a full-featured editing experience with familiar keybindings, mouse support, and multiple cursors. It runs entirely in the terminal while offering capabilities typically found in graphical editors, including split panes, a command palette, and integrated language server protocol support for real-time diagnostics, completions, go-to-definition, and code actions across multiple languages. The editor distinguishes itself through a plugin system that runs sandboxed TypeScript plugins in a QuickJS runtime, with an asynchronous bridge

Este proyecto es un cliente de API PHP y SDK para integrar servicios de OpenAI en aplicaciones PHP. Sirve como una librería de integración y wrapper para interactuar con modelos de lenguaje grandes para generar texto, imágenes y audio mediante llamadas a la API REST. La librería proporciona orquestación especializada para asistentes de IA, gestionando hilos de conversación y almacenes vectoriales. También incluye herramientas para el ajuste fino (fine-tuning) de modelos personalizados, implementación de búsqueda semántica mediante embeddings de texto y procesamiento de audio para transcripción y síntesis. La superficie de capacidad cubre la moderación de contenido, gestión de archivos y el manejo de chats y completado de texto. Soporta la entrega de datos en tiempo real mediante streaming de eventos enviados por el servidor (SSE), procesamiento por lotes asíncrono y la verificación de firmas de webhook.

Spout is a spreadsheet file processing library and multi-format generator designed for reading and writing CSV, XLSX, and ODS files. It functions as a stream-based parser that processes large spreadsheet files incrementally to avoid loading entire documents into memory. The library provides capabilities for programmatic spreadsheet generation and data extraction. It supports custom content styling, allowing for the application of fonts, backgrounds, borders, and number formats to individual cells or rows. Beyond basic file input and output, the project covers workbook manipulation through se

Apollo Android es una biblioteca cliente de GraphQL con tipado fuerte para ejecutar consultas y mutaciones a través de JVM, Android y Kotlin Multiplatform. Incluye una herramienta de generación de código que crea clases inmutables y analizadores a partir de esquemas de GraphQL y archivos de operación para asegurar la seguridad de tipos en tiempo de compilación. El proyecto cuenta con una caché normalizada que aplana las respuestas en una tabla de búsqueda para mantener una única fuente de verdad. También proporciona un cliente de suscripción para actualizaciones de datos del lado del servidor en tiempo real a través de conexiones WebSocket o SSE y un framework de mocking para simular respuestas del servidor y generar datos falsos con tipado fuerte para pruebas. La biblioteca cubre una amplia gama de capacidades, incluyendo optimización de red mediante el procesamiento por lotes de solicitudes y consultas persistentes, así como la gestión del estado de datos locales con actualizaciones optimistas y paginación. Soporta la integración de motores HTTP personalizados y el mapeo de tipos escalares personalizados a clases de lenguaje nativo. La herramienta se integra con IDEs de desarrollo para proporcionar generación automática de código y navegación desde el editor hasta las definiciones.

This project is a PHP SQL REST API generator and database API wrapper that automatically transforms SQL database tables into a functional web interface. It serves as a lightweight layer that maps HTTP methods to SQL commands, allowing for the creation, reading, updating, and deletion of records without writing manual endpoint code. The tool distinguishes itself by providing a dedicated spatial data API gateway for querying and exporting geometry columns using GeoJSON and WKT standards. It also functions as a multi-tenant data API, capable of isolating records for different users through share

tinygltf is a C++ library designed as a parser, serializer, and validation tool for 3D models in the glTF 2.0 binary and JSON formats. It functions as a streaming processor that can handle large 3D asset files incrementally through a callback-driven approach to reduce memory overhead. The library uses a header-only implementation to simplify integration. It employs memory mapping to map binary buffers directly into memory and utilizes a symmetric parse-write pipeline to maintain data consistency between reading and writing files. The system provides capabilities for 3D asset processing, incl

Vocal-separate es una herramienta de procesamiento de audio diseñada para aislar pistas vocales e instrumentales de archivos de audio y video. Funciona como un motor de inteligencia artificial local que realiza la separación de fuentes directamente en la máquina del usuario, asegurando la privacidad de los datos al eliminar la necesidad de conectividad con servidores externos. El sistema proporciona una interfaz de control basada en navegador para gestionar subidas de medios y monitorear tareas de procesamiento. Para manejar la descomposición intensiva de señales, utiliza procesamiento de tensores acelerado por hardware, que descarga cálculos matemáticos complejos a hardware gráfico dedicado para mejorar la velocidad y la eficiencia de la memoria. El software incluye una cola de tareas asíncrona para gestionar múltiples solicitudes secuencialmente y soporta integración programática a través de una interfaz de red local. Esto permite a los usuarios automatizar flujos de trabajo de separación de audio por lotes conectando scripts o aplicaciones personalizadas al motor de procesamiento.