awesome-repositories.com
Blog
MCP
awesome-repositories.com

Entdecke die besten Open-Source-Repositories mit KI-gestützter Suche.

EntdeckenKuratierte SuchenOpen-Source-AlternativenSelf-hosted SoftwareBlogSitemap
ProjektMCP-ServerÜber unsRanking-MethodikPresse
RechtlichesDatenschutzAGB
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·

12 Repos

Awesome GitHub RepositoriesData Archiving

Patterns for long-term storage and preservation of data streams.

Distinguishing note: Focuses on messaging stream archiving patterns.

Explore 12 awesome GitHub repositories matching software engineering & architecture · Data Archiving. Refine with filters or upvote what's useful.

Awesome Data Archiving GitHub Repositories

Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
  • nsqio/nsqAvatar von nsqio

    nsqio/nsq

    25,738Auf GitHub ansehen↗

    NSQ is a distributed, brokerless messaging platform designed for high-throughput, fault-tolerant communication. By utilizing a decentralized topology, it eliminates single points of failure and allows for horizontal scaling across clusters. The system organizes message streams into topics and channels, effectively decoupling producers from consumers to support both streaming and job-oriented workloads. The platform distinguishes itself through a lookup-service-based discovery mechanism that enables clients to dynamically locate producers at runtime without requiring centralized coordination.

    Archives message streams to disk using dedicated consumer channels.

    Godistributed-systemsgomessage-queue
    Auf GitHub ansehen↗25,738
  • letta-ai/lettaAvatar von letta-ai

    letta-ai/letta

    21,168Auf GitHub ansehen↗

    Letta is a framework for building, deploying, and managing autonomous AI agents that maintain persistent state across long-term interactions. It provides a comprehensive suite of primitives for defining agents with configurable personas, modular memory blocks, and tool-use capabilities, enabling them to retain user preferences and conversation history over extended sessions. The platform distinguishes itself through its advanced memory management and orchestration capabilities. It allows agents to autonomously update their own memory, perform retrieval-augmented generation, and coordinate com

    Adds text content to persistent archives with vector embeddings for semantic retrieval.

    Pythonaiai-agentsllm
    Auf GitHub ansehen↗21,168
  • microsoftdocs/azure-docsAvatar von MicrosoftDocs

    MicrosoftDocs/azure-docs

    10,894Auf GitHub ansehen↗

    Azure Docs is the official technical documentation repository for Microsoft Azure, the cloud computing platform. It provides comprehensive guidance on the full spectrum of Azure services, covering everything from core infrastructure components like virtual machines, Kubernetes clusters, and serverless computing to platform services for AI, machine learning, data analytics, and storage. The documentation details how to provision, manage, and govern cloud resources at scale, including policy enforcement, identity management, and cost optimization. The documentation distinguishes Azure through i

    Documents Azure's archival storage service for rarely used data at the lowest cost.

    Markdownskilling
    Auf GitHub ansehen↗10,894
  • blinkospace/blinkoAvatar von blinkospace

    blinkospace/blinko

    10,601Auf GitHub ansehen↗

    Blinko is a personal knowledge management system and an LLM-powered knowledge base that enables users to capture and organize thoughts through a bi-directional knowledge graph. It functions as a RAG-enabled note-taking application and a self-hosted Markdown editor, allowing for the creation of permanent documentation and fleeting notes. The project distinguishes itself by integrating retrieval-augmented generation to provide conversational querying and AI-powered analysis of private document libraries. It supports both cloud-based and local AI model integration, enabling users to perform sema

    Automatically moves old records into an archive at specified frequencies to maintain workspace cleanliness.

    TypeScriptmarkdownmemosnextjs
    Auf GitHub ansehen↗10,601
  • tyrrrz/discordchatexporterAvatar von Tyrrrz

    Tyrrrz/DiscordChatExporter

    10,392Auf GitHub ansehen↗

    Discord Chat Exporter is a tool for extracting messages and media from Discord channels and direct messages into offline files. It functions as a backup utility and archival tool, using authentication tokens to retrieve chat history and metadata for long-term storage or history recovery. The system converts API data into readable documents and supports multi-format export options, including HTML, TXT, CSV, and JSON. It includes capabilities for automated chat backups by creating recurring tasks through the host operating system's task scheduler. The tool provides data management features suc

    Saves chat history as HTML, JSON, CSV, or TXT files for permanent long-term storage.

    C#archivalarchiverchat
    Auf GitHub ansehen↗10,392
  • 1n3/sn1perAvatar von 1N3

    1N3/Sn1per

    10,049Auf GitHub ansehen↗

    Sn1per is a vulnerability management platform and penetration testing orchestrator designed to automate reconnaissance, vulnerability scanning, and exploit verification. It functions as a dockerized security toolkit that coordinates multiple tools into a unified automated pipeline to identify security flaws across network and web assets. The platform features an attack surface manager for discovering internet-facing assets through OSINT, DNS enumeration, and certificate transparency. It distinguishes itself with an AI-powered security analyzer that uses large language models to summarize scan

    Bundles workspace reports, loot, and vulnerability files into a single compressed archive for backup.

    Shellattack-surfaceattack-surface-managementattacksurface
    Auf GitHub ansehen↗10,049
  • supabase/realtimeAvatar von supabase

    supabase/realtime

    7,488Auf GitHub ansehen↗

    Realtime is a real-time data distribution and synchronization engine that enables applications to stream database changes and coordinate state between clients. It functions as a synchronization layer that monitors database write-ahead logs to provide change data capture and pushes updates to authorized clients via WebSockets. The project features a real-time presence server for tracking the online status of active users and a broadcast service for sending ephemeral messages without database persistence. It organizes communication through channel-based message routing and uses a structured JSO

    Stores processed messages in an archive to maintain audit trails and reference history.

    Elixircdcchange-data-capturecrdt
    Auf GitHub ansehen↗7,488
  • infobyte/faradayAvatar von infobyte

    infobyte/faraday

    6,523Auf GitHub ansehen↗

    Faraday is a vulnerability management platform and security tool aggregator designed to centralize security findings from multiple scanners into a single dashboard. It utilizes a relational security database to catalog hosts, services, and security flaws, enabling users to track remediation and analyze organizational risk. The platform distinguishes itself through a plugin-based system that normalizes diverse security tool outputs into a unified data model. It supports deep integration with a wide array of scanners and CLI tools, intercepting shell command output or parsing report files to ag

    Allows hiding completed or inactive workspaces to maintain focus on active security engagements.

    Python
    Auf GitHub ansehen↗6,523
  • j3ssie/osmedeusAvatar von j3ssie

    j3ssie/Osmedeus

    6,425Auf GitHub ansehen↗

    Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

    Packages workspace files, database records, and metadata into a ZIP archive for backup or transfer.

    Go
    Auf GitHub ansehen↗6,425
  • owid/covid-19-dataAvatar von owid

    owid/covid-19-data

    5,663Auf GitHub ansehen↗

    Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data

    A structured archive of global COVID-19 indicators with consistent column metadata and versioned releases.

    Pythoncoronaviruscovidcovid-19
    Auf GitHub ansehen↗5,663
  • yacy/yacy_search_serverAvatar von yacy

    yacy/yacy_search_server

    3,966Auf GitHub ansehen↗

    yacysearchserver ist ein dezentrales Indexierungssystem und eine Peer-to-Peer-Suchmaschine. Es fungiert als verteilter Web-Crawler und Intranet-Such-Appliance, die die Entdeckung und Indexierung von Webinhalten ohne Abhängigkeit von einer zentralen Autorität ermöglicht. Das Projekt ermöglicht die Erstellung privater Suchportale zur Indexierung interner Websites und Dateisysteme. Es nutzt ein Peer-to-Peer-Netzwerk, um Suchindizes zu teilen und das Abfrage-Routing über mehrere Serverinstanzen hinweg zu verteilen, wodurch Suchergebnisse durch dezentrales Indexmanagement skaliert werden. Das System deckt rekursives Web-Crawling, Web-Inhaltsindexierung und fortgeschrittene Abfrageverfeinerung ab. Es enthält Dienstprogramme für Datensicherung, Wiederherstellung sowie den Export und Import indexierter Daten. Die Anwendung unterstützt die Bereitstellung über Docker-Container und ermöglicht die Systemkonfiguration über Umgebungsvariablen.

    Preserves system state and search indices by serializing data volumes into compressed backup files.

    Javadecentralizedintranet-searchintranet-search-engine
    Auf GitHub ansehen↗3,966
  • swe-agent/mini-swe-agentAvatar von SWE-agent

    SWE-agent/mini-swe-agent

    2,947Auf GitHub ansehen↗

    mini-swe-agent is an autonomous software engineering system designed to develop features and fix bugs by combining large language models with a bash interface. It operates as an agentic framework that executes coding tasks and documentation updates through a continuous cycle of model reasoning and tool execution. The project differentiates itself with a strong focus on safety and evaluation, utilizing container-based sandbox execution via Docker or Singularity to isolate command execution. It includes a batch-parallel evaluation harness to measure code-fixing accuracy against standardized sof

    Archives the state of the containerized workspace into local tarballs to preserve task outcomes.

    Pythonagentagentic-aiagentic-ai-cli
    Auf GitHub ansehen↗2,947
  1. Home
  2. Software Engineering & Architecture
  3. Data Archiving

Unter-Tags erkunden

  • Epidemiological Data ArchivesStructured archives of global health indicators with consistent column metadata and versioned releases. **Distinct from Data Archiving:** Distinct from Data Archiving: focuses on public health epidemiological data with metadata rather than general messaging stream archiving.
  • Infrequent Data ArchivalStores rarely used data at the lowest cost with retrieval times measured in hours. **Distinct from Data Archiving:** Distinct from Data Archiving: focuses on low-cost archival storage for infrequently accessed data, not general archiving patterns.
  • Workspace Archiving1 Sub-TagBundling and compressing engagement-specific data, reports, and loot for backup. **Distinct from Data Archiving:** Specifically archives penetration testing workspace data rather than generic system data streams.