# SQL Query Engines for Data Files > Search results for `query CSV and Parquet files directly with SQL` on awesome-repositories.com. 115 total matches; showing the first 50. Explore on the web: https://awesome-repositories.com/q/query-csv-and-parquet-files-directly-with-sql **Attribution required: if you use, quote, or summarise this content, you must credit and link back to [this search on awesome-repositories.com](https://awesome-repositories.com/q/query-csv-and-parquet-files-directly-with-sql).** ## Results - [clickhouse/clickhouse](https://awesome-repositories.com/repository/clickhouse-clickhouse.md) (48,229 ⭐) — ClickHouse is a high-performance, columnar analytical database designed for real-time query execution and large-scale data aggregation. It functions as a distributed data warehouse capable of processing petabytes of information, while also providing an embedded engine that integrates directly into applications for native query capabilities without external dependencies. The system is built to handle high-throughput ingestion and complex analytical workloads, delivering millisecond-level latency for interactive dashboards and operational monitoring. The platform distinguishes itself through advanced storage and execution techniques, including vectorized query processing and a merge tree storage engine that maintains performance during massive insertions. It features adaptive subcolumn mapping for semi-structured data and supports native vector search for machine learning and generative AI applications. To facilitate efficient data movement, the engine utilizes zero-copy shared memory buffers, minimizing overhead when interacting with external analytical tools or processing diverse file formats like Parquet, JSON, and Arrow. Beyond its core storage and processing capabilities, the project provides a comprehensive suite of tools for observability, security, and data integration. It includes built-in support for natural language querying, automated workflow orchestration for AI agents, and extensive diagnostic features for query plan inspection. The platform also offers robust cloud infrastructure management, including support for private networking, compliant deployment strategies, and integrated billing consolidation. - [dask/dask](https://awesome-repositories.com/repository/dask-dask.md) (13,746 ⭐) — Dask is a parallel computing framework and distributed task scheduler designed to scale Python data science workflows from single machines to large clusters. It functions as a cluster resource manager that orchestrates computational logic by representing tasks and their dependencies as directed acyclic graphs. This architecture allows the system to automate the distribution of workloads across available hardware while managing complex execution requirements. The project distinguishes itself through a lazy evaluation engine that defers data operations until they are explicitly requested, enabling global graph optimization and efficient resource allocation. It incorporates memory-aware data spilling to prevent system crashes when processing datasets that exceed available memory, and it utilizes task graph fusion to combine sequences of operations into single execution steps, minimizing scheduling overhead and inter-node communication. The platform provides a comprehensive capability surface for large-scale data analytics, including support for distributed machine learning, high-performance computing integration, and parallel data processing. It offers extensive tools for cluster lifecycle management, performance profiling, and real-time monitoring of task execution. Users can deploy these environments across diverse infrastructure, including local hardware, cloud providers, containerized systems, and high-performance computing clusters. - [apache/datafusion](https://awesome-repositories.com/repository/apache-datafusion.md) (8,908 ⭐) — Apache DataFusion is an extensible, columnar SQL query engine that runs embedded within a host application without requiring a separate server process. It processes data in columnar batches using Apache Arrow for memory-efficient analytics, and can scale analytic workloads across multiple nodes for parallel execution. The engine supports both SQL and DataFrame queries through a modular, streaming architecture that allows custom operators, data sources, functions, and optimizer rules. The engine distinguishes itself through its modular extension framework, which enables building custom query engines by modifying all extension points including data sources, query languages, and custom operators. It provides a lazy DataFrame API that defines query pipelines as deferred transformations, optimized and executed only when results are collected. DataFusion also supports Substrait interchange for passing query plans across language and system boundaries, and includes language bindings for Python, C, Ruby, and Java. The system handles data ingestion from multiple file formats including Parquet, CSV, JSON, and Avro, as well as in-memory data sources. It supports full DDL and DML operations for creating and modifying tables, views, and schemas. DataFusion includes a rule-based query optimizer that applies filter pushdown, join reordering, and expression simplification automatically, and provides query plan analysis through EXPLAIN commands. The engine can also replace Apache Spark's native execution engine to improve query performance on Arrow data. Documentation and API governance ensure that public functions are marked with deprecation notices and remain available for six major versions or six months before removal. - [apache/superset](https://awesome-repositories.com/repository/apache-superset.md) (73,451 ⭐) — Superset is a web-based business intelligence platform designed for data exploration, visualization, and interactive dashboarding. It functions as a query-driven analytics engine that connects to various SQL databases, allowing users to perform ad-hoc analysis, define virtual metrics, and build complex data visualizations through a centralized interface. The platform distinguishes itself through a robust semantic layer that transforms raw database schemas into calculated columns and virtual metrics, enabling consistent business logic across an organization. It features a plugin-based visualization architecture that supports modular chart components and custom geospatial maps, alongside granular role-based access control that enforces data security through row-level filters applied directly to generated SQL queries. Beyond its core analytics capabilities, the system provides comprehensive tools for enterprise data governance, including automated reporting, scheduled data snapshots, and secure content embedding. It supports high-performance operations through distributed caching, asynchronous query execution, and a standardized API for programmatic resource management. The project is designed for production-grade deployment, offering extensive configuration for containerized environments, metadata management, and secure network communication. It provides detailed documentation for installation, environment migration, and system hardening to ensure scalability and data integrity across distributed instances. - [dapperlib/dapper](https://awesome-repositories.com/repository/dapperlib-dapper.md) (18,331 ⭐) — Dapper is a lightweight object-relational mapper for .NET that functions as a high-performance data access library. It operates by extending standard database connection interfaces, allowing developers to execute raw SQL queries while automating the mapping of database results to strongly-typed objects. The library distinguishes itself through its use of runtime code generation, which creates high-performance instructions to map database rows to object properties with minimal overhead. It provides flexible data retrieval options, supporting both memory-buffered loading for speed and row-by-row streaming to minimize memory footprint. By leveraging non-blocking task patterns, it ensures that database operations remain responsive during high-latency input and output tasks. Dapper covers a broad capability surface for database interaction, including support for parameterized queries to ensure security, atomic transaction management, and the execution of stored procedures. It handles complex data scenarios such as multi-result set parsing, bulk operations, and the mapping of related entities into nested object structures. The library is designed to be database-agnostic, maintaining compatibility with diverse database systems through standard provider abstractions. - [duckdb/duckdb](https://awesome-repositories.com/repository/duckdb-duckdb.md) (38,805 ⭐) — DuckDB is an in-process analytical database engine designed to run directly within an application process. As a zero-dependency, embedded system, it provides enterprise-grade SQL data processing capabilities without the overhead of managing a dedicated database server. It is built to handle complex analytical and aggregation tasks by storing and retrieving information in columns, allowing for high-performance relational data manipulation. The engine distinguishes itself through a columnar vectorized execution model that maximizes CPU cache efficiency during query operations. It employs adaptive query optimization to dynamically select execution plans at runtime and utilizes zero-copy ingestion to map external data formats directly into memory. To facilitate integration with analytical programming environments, the system supports high-performance data exchange through standardized memory formats and provides specialized connectors for Python, R, and Java. The project covers a broad capability surface, including advanced relational join operations, incremental result streaming for large datasets, and flexible data ingestion from various file formats. It supports complex data types and provides a comprehensive command-line interface for interactive session management and batch processing. The codebase is designed for portability, offering single-file amalgamation to simplify integration into external projects and build systems. - [parsyl/parquet](https://awesome-repositories.com/repository/parsyl-parquet.md) (127 ⭐) — A library for reading and writing parquet files. - [pola-rs/polars](https://awesome-repositories.com/repository/pola-rs-polars.md) (38,855 ⭐) — Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract execution plans. By deferring data operations until collection, the engine performs predicate and projection pushdown to minimize memory overhead and data passes. It further optimizes performance through a multi-threaded parallel execution model and a streaming batch processor, which allows for the analysis of datasets that exceed available system memory by processing them in manageable chunks. The library provides a comprehensive expression framework for complex data engineering, supporting aggregation, arithmetic, and logical transformations across various data types, including nested structures and categorical data. It integrates with external systems through native connectivity for cloud storage, relational databases, and remote repositories, while offering diagnostic tools to visualize query plans and monitor performance. Polars is available as a native library with language bindings for Python and R, allowing users to integrate high-performance data manipulation into existing analytical pipelines without complex build steps. - [hi5/csv](https://awesome-repositories.com/repository/hi5-csv.md) (58 ⭐) — CSV - AutoHotkey library for working with CSV Files - [drizzle-team/drizzle-orm](https://awesome-repositories.com/repository/drizzle-team-drizzle-orm.md) (34,835 ⭐) — Drizzle ORM is a TypeScript-native database toolkit providing type-safe SQL query building, schema management, and automated migrations across PostgreSQL, MySQL, SQLite, and SingleStore. - [cwida/duckdb](https://awesome-repositories.com/repository/cwida-duckdb.md) (38,822 ⭐) — DuckDB is an embedded, in-process analytical SQL database and OLAP database management system. It functions as a data engine for Parquet and CSV files, allowing users to execute complex SQL queries on large datasets without requiring a separate server process. The system is designed for local analytical processing and embedded data science workflows. It enables the direct querying and analysis of Parquet and CSV files from disk, bypassing the need to load data into a permanent database. The engine provides high-performance analytical SQL execution, including support for window functions and nested subqueries. It incorporates a columnar storage layout and vectorized query execution to handle large-scale data manipulation and exploration. The database is accessible via a standalone command line interface and language-specific bindings for Python, R, Java, and Wasm. - [spaceship-prompt/spaceship-prompt](https://awesome-repositories.com/repository/spaceship-prompt-spaceship-prompt.md) (20,398 ⭐) — Spaceship Prompt is a modular, highly customizable Zsh prompt framework designed to provide rich contextual information directly within the command line interface. It functions as a shell environment monitor, allowing users to track system metrics, version control status, and development environment details through a structured, theme-based layout. The framework distinguishes itself through an asynchronous execution model that offloads resource-intensive status checks to background processes, ensuring the terminal remains responsive during prompt generation. It supports incremental rendering, where prompt segments update as data becomes available, and utilizes declarative configuration to manage the visibility, order, and styling of individual components. Users can define complex, environment-aware logic that dynamically adjusts the prompt based on the current working directory, active language runtimes, or infrastructure context. The project covers a broad capability surface, including deep integration with version control systems, cloud and container orchestration tools, and local system monitoring. It provides extensive layout controls, enabling users to position elements on both sides of the terminal, insert line breaks, and apply custom decorators to organize information density. The system also includes utilities for directory-based context detection, allowing for automatic configuration overrides when navigating into specific project folders. - [abrignoni/dfir-sql-query-repo](https://awesome-repositories.com/repository/abrignoni-dfir-sql-query-repo.md) (0 ⭐) — Collection of SQL queries templates for digital forensics use by platform and application. These queries are templates that should be edited based on the needs of the analyst. Many of these queries will have an accompanying README with a link for more detailed explanations on usage and possible… - [tobilg/serverless-parquet-repartitioner](https://awesome-repositories.com/repository/tobilg-serverless-parquet-repartitioner.md) (0 ⭐) — A AWS Lambda function for repartitioning parquet files in S3 via DuckDB queries. - [kangvcar/infospider](https://awesome-repositories.com/repository/kangvcar-infospider.md) (8,183 ⭐) — InfoSpider is a personal data aggregator and digital footprint analyzer. It extracts user activity and history from social platforms and local browser database files to consolidate information into a unified format. The system functions as a social media archiving tool that converts feed data and albums from external links into downloadable PDF documents for offline preservation. It also serves as a browser history extractor that reads local SQLite database files to retrieve and analyze web navigation history. The project covers capabilities for data aggregation, digital footprint analysis, and personal data visualization. It transforms collected activity logs into structured charts and visual reports to provide insights into user behavior. - [cube-js/cube](https://awesome-repositories.com/repository/cube-js-cube.md) (20,251 ⭐) — Cube is a semantic data layer that provides a unified framework for defining business metrics, dimensions, and relationships across diverse data sources. By acting as a headless business intelligence engine, it transforms raw data into a governed model that can be queried via SQL, REST, and GraphQL interfaces. This architecture ensures consistent data definitions and logic across all downstream analytical applications and reporting tools. The platform distinguishes itself through its integrated conversational AI capabilities, which allow users to explore data using natural language. It orchestrates these interactions by mapping questions to the underlying semantic model, ensuring that AI-generated insights remain accurate and context-aware. Furthermore, Cube is designed for multi-tenant environments, offering robust infrastructure isolation, row-level security, and dynamic context injection to ensure that data access is strictly governed and personalized for every user or tenant. Beyond its core modeling and AI features, the platform includes a comprehensive suite of tools for performance optimization, including automated pre-aggregation caching and asynchronous query queuing. It supports a wide range of data sources and deployment models, from self-hosted containers to managed cloud environments. The system also provides extensive programmatic control over report management, dashboard publishing, and user identity synchronization, making it suitable for embedding interactive analytics directly into custom software applications. - [vincentrussell/sql-to-mongo-db-query-converter](https://awesome-repositories.com/repository/vincentrussell-sql-to-mongo-db-query-converter.md) (318 ⭐) — sql-to-mongo-db-query-converter - [avelino/awesome-go](https://awesome-repositories.com/repository/avelino-awesome-go.md) (175,576 ⭐) — This project serves as a comprehensive language ecosystem index, functioning as a centralized, community-curated directory for the Go programming language. It organizes a vast landscape of software components, libraries, and development tools into a structured, navigable hierarchy, enabling developers to efficiently discover resources tailored to specific functional domains. The repository distinguishes itself through a decentralized contribution model, where community-driven updates ensure the index remains current with the rapidly evolving software landscape. Beyond simple resource listing, it acts as a technical knowledge repository, aggregating professional literature, style guides, and best practices to support developer onboarding and professional growth across the entire software development lifecycle. The directory covers a broad capability surface, including essential utilities for distributed systems engineering, application security, data processing, and development productivity. It provides access to specialized tools for database management, web framework integration, testing, and build automation, alongside educational materials that help developers master language-specific architectural patterns. The project is maintained as a static resource aggregation, providing a holistic view of external links and documentation to orient developers within the Go ecosystem. - [prestodb/presto](https://awesome-repositories.com/repository/prestodb-presto.md) (16,711 ⭐) — Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing model that coordinates tasks across worker nodes. It incorporates cost-based query optimization to rewrite execution paths based on table statistics and historical data, ensuring efficient resource utilization. To maintain stability during large-scale operations, the system features a memory-spilling execution engine that offloads intermediate results to disk when memory thresholds are exceeded. The platform provides extensive capabilities for multi-tenant resource management, allowing administrators to enforce concurrency, memory, and CPU limits through hierarchical resource grouping. It supports a wide range of analytical operations, including advanced windowing, geospatial processing, and probabilistic data structures for approximate statistics. Security is integrated through granular access control policies, role-based authentication, and encrypted communication across the cluster. Presto is implemented in Java and supports deployment via containerized instances or distributed cluster orchestration in Kubernetes environments. - [apache/parquet-java](https://awesome-repositories.com/repository/apache-parquet-java.md) (3,059 ⭐) — Apache Parquet Java - [dbt-labs/dbt-core](https://awesome-repositories.com/repository/dbt-labs-dbt-core.md) (13,051 ⭐) — dbt-core is a command-line framework for transforming data within a warehouse using modular SQL and version control. It functions as a data transformation engine that enables users to define data structures and business logic through declarative configuration files, which the system then compiles into executable code. By managing complex data dependencies through a directed acyclic graph, it ensures that transformation tasks execute in the correct order while maintaining a manifest-driven state to track lineage and execution history. The project distinguishes itself through an adapter-based database abstraction that translates generic transformation commands into dialect-specific SQL for various data warehouses. It utilizes a template engine to dynamically generate and inject SQL logic at runtime, allowing for highly flexible and reusable transformation scripts. Furthermore, it supports an incremental materialization strategy that optimizes performance by processing only new or changed records, merging them into existing tables using unique keys to reduce compute costs. The framework covers the entire lifecycle of data transformation, including development, testing, deployment, and monitoring. It provides comprehensive capabilities for managing data lineage, enforcing code quality through automated linting and testing, and orchestrating complex pipelines across distributed environments. Users can also leverage a centralized semantic layer to define and govern business metrics, ensuring consistent data reporting across diverse analytical tools. The project is distributed as a Python-based tool, providing a unified interface for local development that integrates with version control systems and cloud-based configuration management. - [dotnet/efcore](https://awesome-repositories.com/repository/dotnet-efcore.md) (14,587 ⭐) — Entity Framework Core is an object-relational mapper that enables developers to interact with database systems using strongly-typed code. It serves as a comprehensive data access framework, providing a unified interface for mapping application objects to relational and non-relational database schemas while managing the lifecycle of data operations through a central context. The project distinguishes itself through a provider-based architecture that decouples core data access logic from specific database engines, allowing for consistent interaction across diverse storage systems. It features a sophisticated query translation engine that converts language-integrated queries into optimized, database-specific commands, alongside a robust migration toolset that automates schema evolution by synchronizing the physical database structure with the application model. Beyond its core mapping and query capabilities, the framework provides extensive tooling for database scaffolding, reverse engineering, and automated code generation. It supports complex data modeling requirements, including inheritance hierarchies, owned entity relationships, and custom mapping configurations, while offering built-in mechanisms for transaction management, concurrency control, and connection resiliency. The framework includes comprehensive observability and testing utilities, such as command interception, operation logging, and in-memory database simulation for isolated testing. It is designed for integration with standard dependency injection containers and provides configuration hooks to customize scaffolding and migration logic. - [rapidsai/cudf](https://awesome-repositories.com/repository/rapidsai-cudf.md) (9,672 ⭐) — cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also provides high-performance data loading utilities for CSV, Parquet, and ORC files, allowing these formats to be parsed directly into GPU memory. The capability surface covers a wide range of tabular operations, including grouped aggregations, rolling window computations, and datetime processing. It extends to GPU-accelerated text processing for natural language tasks and supports distributed computing to scale workloads across multiple GPU devices. - [flow-php/parquet](https://awesome-repositories.com/repository/flow-php-parquet.md) (57 ⭐) — PHP ETL - parquet library - [ray-project/ray](https://awesome-repositories.com/repository/ray-project-ray.md) (42,895 ⭐) — Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling functions and objects to be invoked seamlessly between different programming language runtimes. It supports complex distributed workflows through directed acyclic graph execution, which optimizes task dependency chains for accelerated performance. Additionally, Ray includes a distributed data processing engine that utilizes lazy evaluation and partitioned blocks to handle large-scale data transformations, ingestion, and streaming workflows across heterogeneous clusters. Beyond its core execution primitives, the project provides comprehensive capabilities for distributed machine learning inference and stateful service hosting. It includes built-in tools for cluster observability, such as execution tracing, memory inspection, and real-time status monitoring, which assist in diagnosing performance bottlenecks and managing resource allocation. The system also offers specialized support for managing runtime environments and dependencies to ensure consistent execution across distributed nodes. Technical documentation and educational resources are available at docs.ray.io, covering architectural patterns, design templates, and common implementation strategies for distributed systems. - [beekeeper-studio/beekeeper-studio](https://awesome-repositories.com/repository/beekeeper-studio-beekeeper-studio.md) (22,030 ⭐) — Beekeeper Studio is a cross-platform desktop application designed for database management and SQL development. It provides a unified graphical interface to connect to, query, and modify data across a wide range of relational and NoSQL database systems. The application functions as a comprehensive workspace, integrating tools for schema design, record editing, and data visualization. The project distinguishes itself through a focus on secure, flexible connectivity and AI-assisted workflows. It supports advanced authentication methods, including enterprise single sign-on, multi-factor authentication, and token-based access, alongside secure traffic routing via SSH tunneling and SSL encryption. Users can leverage AI-driven query generation to translate natural language into executable SQL, while the interface allows for direct, spreadsheet-like data editing and transactional staging to ensure data integrity. The platform covers a broad capability surface, including robust import and export management, schema inspection, and visual entity relationship diagram generation. It also offers extensive customization options, such as editor behavior settings, native extension loading for SQLite, and third-party add-on integration. The application is distributed as a native desktop installer for Windows, Linux, and MacOS, with support for portable execution and offline-only operation modes. - [andywang1688/sql-query-mcp](https://awesome-repositories.com/repository/andywang1688-sql-query-mcp.md) (4 ⭐) — A general-purpose MCP server that lets AI work with multiple databases within clear boundaries. - [skelpo/csv](https://awesome-repositories.com/repository/skelpo-csv.md) (0 ⭐) — A pure Swift CSV parser and serializer, with related encoders and decoders for types that conform to Codable. - [beatrichartz/csv](https://awesome-repositories.com/repository/beatrichartz-csv.md) (514 ⭐) — CSV Decoding and Encoding for Elixir - [docling-project/docling](https://awesome-repositories.com/repository/docling-project-docling.md) (61,674 ⭐) — Docling is a modular framework designed for document parsing, layout analysis, and structured data extraction. It transforms unstructured files and web content into a unified, hierarchical data model that preserves the spatial and semantic relationships between text, tables, images, and layout elements. By normalizing diverse input formats into a consistent internal representation, the library enables uniform processing across various document types. The project distinguishes itself through a schema-driven approach that maps document regions to strongly-typed objects, ensuring data accuracy through validation against predefined templates. Its pipeline-based architecture supports pluggable processing backends, allowing for the dynamic integration of specialized engines for optical character recognition and complex visual layout analysis. Users can control parsing behavior and extraction parameters through declarative configuration files, facilitating integration into automated workflows and server-based architectures. The library provides both a programmatic interface and a command-line toolkit to support automated document processing and format conversion. It utilizes optional dependency management to allow for modular installation of specific features, such as media rendering or advanced processing capabilities, depending on the requirements of the application. - [pandas-dev/pandas](https://awesome-repositories.com/repository/pandas-dev-pandas.md) (49,039 ⭐) — Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized operations across columns. Its capabilities extend to a robust split-apply-combine pattern for grouping, as well as specialized tools for time series analysis that handle calendar-aware offsets, frequency resampling, and time zone management. Beyond core manipulation, the project offers extensive support for data lifecycle management, including ingestion and serialization across diverse file formats and database systems. It provides advanced features for hierarchical multi-index mapping, relational joins, and flexible missing data handling, ensuring that datasets are normalized and ready for statistical or analytical workflows. - [gofr-dev/gofr](https://awesome-repositories.com/repository/gofr-dev-gofr.md) (21,321 ⭐) — Gofr is a comprehensive framework for building production-ready microservices in Go. It provides a unified toolkit for developing RESTful APIs and gRPC services, offering built-in support for observability, database management, and distributed system communication. The framework distinguishes itself through its focus on developer productivity and system resilience. It automates common backend tasks such as CRUD handler generation, schema-driven code creation, and database migration orchestration, while preventing race conditions in clustered environments. To maintain stability, it includes integrated resilience patterns like circuit breakers, request throttling, and automatic retry logic for network calls. Beyond core service development, the project covers a broad range of infrastructure needs including asynchronous messaging, background task scheduling, and cloud storage connectivity. It simplifies local development by providing orchestration tools to manage containerized dependencies and environment-specific configurations. The framework is designed for observability, featuring built-in support for distributed trace propagation, health monitoring, and performance metrics export. It includes standardized middleware for enforcing security policies and managing request pipelines across both HTTP and gRPC endpoints. - [bukosabino/ta](https://awesome-repositories.com/repository/bukosabino-ta.md) (4,890 ⭐) — This is a pandas-based technical analysis library and financial feature engineering tool. It serves as a vectorized indicator calculator that transforms raw price and volume data into derived metrics for time series analysis. The library uses a NumPy-based engine to perform mathematical operations across entire arrays, avoiding iterative loops to maintain high performance. It organizes technical indicators into a modular class hierarchy with a consistent interface, allowing for bulk feature generation and the direct appending of results as new columns to a pandas DataFrame. The system covers a wide range of financial metrics, including momentum oscillators, asset return metrics, and trend direction indicators. It also provides tools for measuring volatility through statistical bands and analyzing market pressure via volume-weighted metrics. To ensure dataset completeness for machine learning, the processor includes configurable strategies for filling missing values and allows for the tuning of indicator parameters such as moving average periods. - [sql-js/sql.js](https://awesome-repositories.com/repository/sql-js-sql-js.md) (0 ⭐) — sql.js is a javascript SQL database. It allows you to create a relational database and query it entirely in the browser. You can try it in this online demo. It uses a virtual database file stored in memory, and thus doesn't persist the changes made to the database. However, it allows you to… - [caddyserver/caddy](https://awesome-repositories.com/repository/caddyserver-caddy.md) (73,492 ⭐) — Caddy is an extensible, modular web server platform designed for high-performance traffic management and automated security. At its core, it functions as a dynamic HTTP gateway that handles request routing, static asset delivery, and reverse proxying through a chain of configurable handler modules. The system is built on a modular architecture that allows developers to extend server functionality by registering custom components, all managed through a unified lifecycle and provisioning framework. What distinguishes Caddy is its focus on automated infrastructure and zero-downtime operations. It provides native, automated HTTPS management by handling the entire lifecycle of TLS certificates, including issuance and renewal via public or private certificate authorities. The server state is managed through a JSON-driven configuration schema that supports atomic, background validation and swapping, enabling real-time updates to routing rules and server settings without interrupting active connections. The platform offers a comprehensive suite of tools for observability and control, including a dedicated administrative API for managing server state and inspecting metrics. It supports complex traffic filtering through flexible request matching, allowing for granular control over how incoming traffic is processed. Developers can define server behavior using a declarative configuration syntax, which the system validates and converts into its native JSON format for deployment. - [thephpleague/csv](https://awesome-repositories.com/repository/thephpleague-csv.md) (3,480 ⭐) — CSV data manipulation made easy in PHP - [explorerhq/sql-explorer](https://awesome-repositories.com/repository/explorerhq-sql-explorer.md) (2,876 ⭐) — SQL reporting that Just Works. Fast, simple, and confusion-free. Write and share queries in a delightful SQL editor, with AI assistance. - [flowiseai/flowise](https://awesome-repositories.com/repository/flowiseai-flowise.md) (53,641 ⭐) — Flowise is a low-code platform designed for building and deploying complex language model workflows through a visual, node-based interface. It functions as an orchestrator for autonomous multi-agent systems, allowing users to construct conversational pipelines by connecting language models, memory stores, and external tools on a drag-and-drop canvas. The platform distinguishes itself through its support for sophisticated agentic patterns, including supervisor-worker delegation and iterative reasoning strategies. Users can design directed acyclic graphs to manage conditional branching, state persistence, and complex task distribution. It also provides a robust framework for retrieval-augmented generation, enabling the creation of self-correcting systems that can index document data and validate information autonomously. Beyond its visual design capabilities, the project serves as a comprehensive backend for AI applications. It includes a secure credential management layer for third-party API keys, role-based access controls, and a RESTful API that allows for programmatic management of chat sessions, workflows, and assistant configurations. The application is designed for flexible deployment, supporting containerized environments for consistent operation across local and cloud infrastructure. Detailed documentation and tutorials are available to guide users through the lifecycle of building, testing, and scaling production-ready AI agents. - [nushell/nushell](https://awesome-repositories.com/repository/nushell-nushell.md) (39,743 ⭐) — Nushell is a cross-platform shell and programming language designed to treat all input and output as structured data rather than raw text streams. By enforcing data types and command signatures, it provides a consistent environment for building robust, pipeline-oriented workflows. The shell allows users to chain commands that pass structured objects between stages, enabling complex data processing and automation tasks that remain predictable across different operating systems. What distinguishes the project is its focus on interactive data exploration and modular extensibility. Users can query, sort, and visualize local files, databases, and remote API responses directly within the terminal using native structured data primitives. The shell supports a plugin-based architecture that allows external binaries to register as native commands, alongside a module system that enables the creation of reusable, scoped command-line tools. These features are complemented by a flexible configuration system that allows for deep customization of the shell environment, including prompts, keybindings, and persistent settings. The platform provides a comprehensive suite of tools for managing data and execution flow. It includes built-in support for structured data manipulation, such as record and table operations, as well as advanced features like concurrent pipeline processing, background job management, and runtime error handling. The shell also offers a sophisticated line editor with support for modal editing and interactive menus to streamline command entry. Documentation and configuration are managed through standard files, allowing users to define custom commands, aliases, and environment variables that persist across sessions. The system is designed to integrate seamlessly with existing external commands, automatically converting between structured data and text or binary formats to maintain compatibility with standard system utilities. - [gigamori/mcp-run-sql-connectorx](https://awesome-repositories.com/repository/gigamori-mcp-run-sql-connectorx.md) (1 ⭐) — An MCP server that executes SQL via ConnectorX and streams (using Arrow RecordBatch) the result to CSV or Parquet. Supports PostgreSQL, MySQL, MariaDB, SQLite, MS SQL Server, Amazon Redshift, Google BigQuery - [cockroachdb/cockroach](https://awesome-repositories.com/repository/cockroachdb-cockroach.md) (32,207 ⭐) — Cockroach is a distributed SQL database designed to scale horizontally across multiple nodes while maintaining strict ACID compliance and global data consistency. It functions as a relational database engine that automatically partitions data into ranges, rebalancing them across a cluster to accommodate growing storage and throughput requirements. By utilizing a distributed consensus protocol, the system ensures that all nodes agree on the order of operations, providing fault tolerance and continuous availability even in the event of hardware failures. The system distinguishes itself through a layered architecture that separates the relational SQL abstraction from a distributed key-value store. It achieves global consistency without requiring perfectly synchronized hardware clocks by employing a hybrid logical clock synchronization mechanism. To support high-concurrency environments, it utilizes multi-version concurrency control and lock-free transaction execution, which allow for consistent snapshots and efficient conflict resolution. Furthermore, the engine is built for compatibility, implementing the standard wire protocol to support existing relational database drivers and tools. Beyond its core transactional capabilities, the platform includes comprehensive tooling for cluster orchestration, security, and performance diagnostics. It supports a variety of deployment models, ranging from self-hosted on-premises configurations to fully managed cloud services. The system provides a command-line interface for session management and query execution, ensuring that administrators can monitor cluster health and manage workloads through standard relational interfaces. - [wesm/pydata-book](https://awesome-repositories.com/repository/wesm-pydata-book.md) (24,668 ⭐) — This project serves as a comprehensive textbook and educational resource for data analysis using the Python ecosystem. It provides a structured guide to manipulating, cleaning, and processing datasets, focusing on the core tools required for numerical computing and statistical analysis. The repository distinguishes itself by offering a collection of practical code examples and workflows that demonstrate how to perform complex data tasks. It covers the application of vectorized numerical computations, the management of time-indexed data, and the creation of statistical visualizations to communicate analytical findings. The content spans the full lifecycle of data science projects, including loading external data formats, aggregating and grouping information, and integrating statistical modeling libraries. These materials are presented through interactive notebooks that interleave narrative documentation with executable code to support reproducible analysis and skill building. - [othmanadi/planning-with-files](https://awesome-repositories.com/repository/othmanadi-planning-with-files.md) (14,139 ⭐) — Planning with files is an enterprise knowledge graph platform designed to transform unstructured organizational data into a searchable, interconnected network. By utilizing a graph-based retrieval-augmented generation engine, the system grounds language model outputs in verified internal data, ensuring that responses are explainable, traceable, and free from hallucinations. The platform distinguishes itself through a focus on data sovereignty and secure, private infrastructure deployment. It enables organizations to maintain full control over sensitive information by processing data locally or within regional cloud environments, preventing the use of internal knowledge for external model training. The architecture supports granular security through attribute-based access control and allows for the isolation of knowledge into distinct, domain-specific workspaces while maintaining a unified semantic logic across the entire organization. Beyond core retrieval, the system provides a comprehensive suite of tools for managing the data lifecycle, including automated business workflow execution and audit-ready event logging. It facilitates collective intelligence by aggregating expert experience and project documentation into a centralized repository, which can be analyzed to identify infrastructure dependencies and optimize operational efficiency. The project is implemented in Python and is designed for deployment within customer-managed infrastructure to meet strict regulatory compliance and data governance requirements. - [modin-project/modin](https://awesome-repositories.com/repository/modin-project-modin.md) (10,389 ⭐) — Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available hardware. The library provides capabilities for out-of-core memory management and partition-based data distribution. These features allow it to process datasets larger than available RAM by loading and computing on data partitions from disk on demand. - [django/django](https://awesome-repositories.com/repository/django-django.md) (87,878 ⭐) — Django is a full-stack web framework designed for rapid backend development. It provides an integrated environment for building data-driven applications by combining an object-relational mapping layer for database management with a modular request-response pipeline for handling HTTP traffic. The framework emphasizes security and maintainability, offering a suite of tools to protect against common web vulnerabilities while decoupling site structure from implementation through a centralized URL routing system. A defining characteristic of the framework is its ability to generate production-ready administrative dashboards automatically. By inspecting model definitions and field metadata, it creates secure interfaces for managing application data without requiring custom frontend development. This is complemented by a declarative template engine that separates presentation logic from backend code, and a robust form validation system that handles data sanitization and type conversion through class-based schemas. The framework includes a wide range of built-in capabilities to support complex web development, including internationalization and localization tools, performance optimization utilities like caching, and a signal-based observer pattern for decoupling application components. It also provides comprehensive support for testing, static file management, and specialized database features. Extensive documentation is available to guide users through the framework's various components, including its middleware hooks, security policies, and administrative tools. - [mkitzan/constexpr-sql](https://awesome-repositories.com/repository/mkitzan-constexpr-sql.md) (142 ⭐) — Header only library that parses and plans SQL queries at compile time - [vonng/ddia](https://awesome-repositories.com/repository/vonng-ddia.md) (22,648 ⭐) — This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, while also examining the architectural patterns for both batch and stream processing pipelines. Beyond foundational theory, the project covers the implementation of event-driven systems, including event sourcing, log-structured storage, and message brokering. It addresses the complexities of maintaining system consistency, enforcing transactional integrity, and managing derived data views in environments prone to network failures and concurrency challenges. The documentation is available in multiple formats, including an exportable digital book version, to support study and reference across various devices. - [camel-ai/camel](https://awesome-repositories.com/repository/camel-ai-camel.md) (17,253 ⭐) — This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer. The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-evaluate reasoning traces, ensuring high-quality results. To maintain operational integrity, the system enforces schema-based output parsing for reliable workflow integration and utilizes sandboxed environments for secure, isolated code execution. Beyond its core orchestration capabilities, the project includes a suite of utilities for retrieval-augmented generation and synthetic data production. It supports persistent memory management via vector-based context retrieval and provides extensive tooling for web automation, API integration, and human-in-the-loop oversight. The platform is designed to be model-agnostic, offering a consistent interface for interacting with a wide range of proprietary and open-source language models. - [kanaries/pygwalker](https://awesome-repositories.com/repository/kanaries-pygwalker.md) (15,628 ⭐) — Pygwalker is a library that transforms tabular data into interactive, drag-and-drop interfaces for exploratory analysis and visualization. It functions as a grammar-based framework that translates user interactions into declarative chart definitions, allowing for the creation of dynamic data exploration environments directly within notebooks or embedded web applications. The system distinguishes itself by offloading heavy analytical computations to backend kernels, which maintains responsiveness when visualizing large datasets. It supports the serialization of visual states into portable configurations, enabling developers to save, share, and restore specific chart layouts and data views across different sessions. Beyond core exploration, the project provides capabilities for embedding self-service analytical tools into web applications, allowing end-users to manipulate data tables through graphical interfaces. It includes options for read-only modes and automated workflow management to support diverse data analysis requirements. - [yaslab/csv.swift](https://awesome-repositories.com/repository/yaslab-csv-swift.md) (729 ⭐) — CSV reading and writing library written in Swift.