12 dépôts
Patterns for long-term storage and preservation of data streams.
Distinguishing note: Focuses on messaging stream archiving patterns.
Explore 12 awesome GitHub repositories matching software engineering & architecture · Data Archiving. Refine with filters or upvote what's useful.
NSQ is a distributed, brokerless messaging platform designed for high-throughput, fault-tolerant communication. By utilizing a decentralized topology, it eliminates single points of failure and allows for horizontal scaling across clusters. The system organizes message streams into topics and channels, effectively decoupling producers from consumers to support both streaming and job-oriented workloads. The platform distinguishes itself through a lookup-service-based discovery mechanism that enables clients to dynamically locate producers at runtime without requiring centralized coordination.
Archives message streams to disk using dedicated consumer channels.
Letta is a framework for building, deploying, and managing autonomous AI agents that maintain persistent state across long-term interactions. It provides a comprehensive suite of primitives for defining agents with configurable personas, modular memory blocks, and tool-use capabilities, enabling them to retain user preferences and conversation history over extended sessions. The platform distinguishes itself through its advanced memory management and orchestration capabilities. It allows agents to autonomously update their own memory, perform retrieval-augmented generation, and coordinate com
Adds text content to persistent archives with vector embeddings for semantic retrieval.
Azure Docs is the official technical documentation repository for Microsoft Azure, the cloud computing platform. It provides comprehensive guidance on the full spectrum of Azure services, covering everything from core infrastructure components like virtual machines, Kubernetes clusters, and serverless computing to platform services for AI, machine learning, data analytics, and storage. The documentation details how to provision, manage, and govern cloud resources at scale, including policy enforcement, identity management, and cost optimization. The documentation distinguishes Azure through i
Documents Azure's archival storage service for rarely used data at the lowest cost.
Blinko is a personal knowledge management system and an LLM-powered knowledge base that enables users to capture and organize thoughts through a bi-directional knowledge graph. It functions as a RAG-enabled note-taking application and a self-hosted Markdown editor, allowing for the creation of permanent documentation and fleeting notes. The project distinguishes itself by integrating retrieval-augmented generation to provide conversational querying and AI-powered analysis of private document libraries. It supports both cloud-based and local AI model integration, enabling users to perform sema
Automatically moves old records into an archive at specified frequencies to maintain workspace cleanliness.
Discord Chat Exporter is a tool for extracting messages and media from Discord channels and direct messages into offline files. It functions as a backup utility and archival tool, using authentication tokens to retrieve chat history and metadata for long-term storage or history recovery. The system converts API data into readable documents and supports multi-format export options, including HTML, TXT, CSV, and JSON. It includes capabilities for automated chat backups by creating recurring tasks through the host operating system's task scheduler. The tool provides data management features suc
Saves chat history as HTML, JSON, CSV, or TXT files for permanent long-term storage.
Sn1per is a vulnerability management platform and penetration testing orchestrator designed to automate reconnaissance, vulnerability scanning, and exploit verification. It functions as a dockerized security toolkit that coordinates multiple tools into a unified automated pipeline to identify security flaws across network and web assets. The platform features an attack surface manager for discovering internet-facing assets through OSINT, DNS enumeration, and certificate transparency. It distinguishes itself with an AI-powered security analyzer that uses large language models to summarize scan
Bundles workspace reports, loot, and vulnerability files into a single compressed archive for backup.
Realtime is a real-time data distribution and synchronization engine that enables applications to stream database changes and coordinate state between clients. It functions as a synchronization layer that monitors database write-ahead logs to provide change data capture and pushes updates to authorized clients via WebSockets. The project features a real-time presence server for tracking the online status of active users and a broadcast service for sending ephemeral messages without database persistence. It organizes communication through channel-based message routing and uses a structured JSO
Stores processed messages in an archive to maintain audit trails and reference history.
Faraday is a vulnerability management platform and security tool aggregator designed to centralize security findings from multiple scanners into a single dashboard. It utilizes a relational security database to catalog hosts, services, and security flaws, enabling users to track remediation and analyze organizational risk. The platform distinguishes itself through a plugin-based system that normalizes diverse security tool outputs into a unified data model. It supports deep integration with a wide array of scanners and CLI tools, intercepting shell command output or parsing report files to ag
Allows hiding completed or inactive workspaces to maintain focus on active security engagements.
Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp
Packages workspace files, database records, and metadata into a ZIP archive for backup or transfer.
Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data
A structured archive of global COVID-19 indicators with consistent column metadata and versioned releases.
yacysearchserver est un système d'indexation décentralisé et un moteur de recherche peer-to-peer. Il fonctionne comme un crawler web distribué et un appareil de recherche intranet qui permet la découverte et l'indexation de contenu web sans dépendance à une autorité centrale. Le projet permet la création de portails de recherche privés pour indexer des sites web internes et des systèmes de fichiers. Il utilise un réseau peer-to-peer pour partager des index de recherche et distribuer le routage des requêtes à travers plusieurs instances de serveur, mettant à l'échelle les résultats de recherche via une gestion d'index décentralisée. Le système couvre le crawling web récursif, l'indexation de contenu web et l'affinement avancé des requêtes. Il inclut des utilitaires pour la sauvegarde, la restauration et l'exportation et l'importation de données indexées. L'application prend en charge le déploiement via des conteneurs Docker et permet la configuration du système via des variables d'environnement.
Preserves system state and search indices by serializing data volumes into compressed backup files.
mini-swe-agent is an autonomous software engineering system designed to develop features and fix bugs by combining large language models with a bash interface. It operates as an agentic framework that executes coding tasks and documentation updates through a continuous cycle of model reasoning and tool execution. The project differentiates itself with a strong focus on safety and evaluation, utilizing container-based sandbox execution via Docker or Singularity to isolate command execution. It includes a batch-parallel evaluation harness to measure code-fixing accuracy against standardized sof
Archives the state of the containerized workspace into local tarballs to preserve task outcomes.