Awesome GitHub RepositoriesPrivate Data Processing Environments

Shannon is an integrated security platform designed for autonomous penetration testing, static and dynamic analysis, and automated vulnerability remediation within self-hosted, private infrastructure. It functions as a unified security suite that orchestrates the entire lifecycle of vulnerability management, from initial discovery and reachability prioritization to the generation and verification of code-level patches. The platform distinguishes itself through its agentic approach to security, deploying autonomous agents to execute both black-box and white-box exploits against running applica

This repository serves as a comprehensive research platform and toolkit for advancing machine learning, quantum computing, and large-scale scientific data analysis. It provides foundational frameworks for developing complex algorithmic systems, offering the necessary infrastructure for distributed training, computational graph execution, and high-performance model development. The project distinguishes itself by integrating specialized research domains with robust, privacy-preserving methodologies. It supports diverse scientific discovery through tools for quantum simulation, physics-informed

Marker is a comprehensive document processing platform designed to automate the conversion, extraction, and structuring of data from complex files. It functions as an orchestration engine that chains modular processing steps into versioned, reusable pipelines, allowing organizations to standardize document handling and automate repetitive business tasks at scale. The platform distinguishes itself through its support for secure, private infrastructure deployment, enabling users to run containerized services within their own environments to maintain strict data privacy. It features specialized

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

DocsGPT is a retrieval-augmented generation platform and private knowledge base used to build AI agents that perform grounded search and analysis. It functions as a multi-model AI orchestrator and enterprise agent builder, allowing for the integration of various local and cloud language models to customize reasoning and text generation. The project provides a visual environment for developing automated assistants using conditional logic and third-party API connectivity. It enables the creation of private AI agents capable of performing enterprise search and detailed document analysis using pr

Unstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t

Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Snyk is an application security testing platform designed to identify and remediate vulnerabilities across source code, open-source dependencies, container images, and infrastructure-as-code configurations. It functions as a comprehensive security workflow automation tool, utilizing a static analysis engine and dependency graph mapping to detect security flaws and license compliance issues throughout the software development lifecycle. The platform distinguishes itself through agentic workflow orchestration and an automated remediation pipeline that generates and submits pull requests to patc

The Snyk CLI is a command-line security scanner that detects known vulnerabilities across open-source dependencies, proprietary application code, container images, and infrastructure-as-code configuration files. It also serves as a platform management tool, allowing users to configure organizations, users, SSO, and reporting from the terminal rather than the web dashboard. The CLI integrates directly into development workflows, enabling scanning within IDEs, build pipelines, and version control systems. It implements static analysis with interfile data flow analysis to find complex security f

Costrict este un agent AI de inginerie software și asistent de codare conceput pentru dezvoltare la nivel enterprise. Funcționează ca un orchestrator AI multi-model care generează, completează și revizuiește codul, servind în același timp ca un mediu de dezvoltare remote care conectează interfețele browserului cu directoare remote pentru gestionarea fișierelor și execuția în terminal. Platforma se distinge printr-un sistem de revizuire a codului AI care utilizează verificare multi-model și indexarea repository-urilor pentru a asigura calitatea codului. Utilizează o abordare structurată de tip agent care descompune cerințele complexe în limbaj natural în fluxuri de lucru secvențiale de analiză, planificare și testare pentru a menține controlul arhitectural. Sistemul acoperă domenii largi de capabilități, inclusiv gestionarea spațiului de lucru remote, integrarea de modele AI personalizate și revizuirea automată a codului pentru repository-uri și git diffs. De asemenea, oferă extensibilitate printr-un sistem de agenți bazat pe competențe și integrare cu instrumente externe printr-un protocol de context standardizat. Proiectul este implementat în TypeScript și oferă o integrare în editor bazată pe plugin-uri pentru a standardiza fluxurile de lucru în editoarele de cod suportate.

Doxx is a collection of shell-based tools for parsing, viewing, and converting word processing files. It provides a terminal user interface for reading and searching DOCX files without requiring external office software, functioning as both a terminal document viewer and a command line parser. The project distinguishes itself by offering a full TUI document reader with outline navigation and the ability to export document content and metadata into alternative formats such as Markdown, CSV, JSON, and plain text. The system covers a broad range of capabilities including document metadata analy

Starred is a utility that automates the management and documentation of starred repositories. It functions by fetching repository metadata through the GitHub API and organizing these projects into structured, categorized lists based on programming language or topic. The tool distinguishes itself by maintaining these lists through automated, scheduled workflows that synchronize data directly to a dedicated repository. It supports the inclusion of private repositories in the generated output, ensuring that a user's complete collection is documented and backed up. The project provides a configu

Acest proiect este un instrument privat de analiză a documentelor care permite interacțiunea conversațională cu fișiere PDF prin executarea întregii inferențe și procesări a modelului de limbaj în întregime pe mașina locală. Prin rularea modelelor direct în browser sau în mediul local, asigură că datele sensibile ale utilizatorului rămân offline și inaccesibile serverelor externe sau furnizorilor de cloud terți. Sistemul utilizează RAG (Retrieval Augmented Generation) pentru a oferi răspunsuri conștiente de context, susținute de extracția textului din documentele locale și indexarea vector embedding. Această arhitectură permite căutarea semantică și regăsirea informațiilor fără a se baza pe servicii de baze de date externe sau conectivitate la internet. Dincolo de capabilitățile conversaționale de bază, instrumentul include funcționalități de observabilitate care loghează pașii interni ai raționamentului modelului și ai lanțurilor de regăsire. Acest tracing al execuției permite depanarea problemelor de performanță și optimizarea calității răspunsurilor în timpul procesului de analiză a documentelor.