Awesome GitHub RepositoriesContent-Based Deduplication

Noms is a distributed version control database and content-addressable data store. It identifies data by cryptographic hashes to ensure integrity and deduplication, while tracking dataset state changes through a sequence of immutable commits to enable branching, forking, and historical recovery. The system functions as a peer-to-peer data synchronizer, reconciling state between disconnected database instances to ensure all nodes converge on the same data. It distinguishes itself as a schema-flexible document store that supports self-describing types, allowing schemas to evolve and widen as ne

Horizon ist ein KI-gestütztes Nachrichtenaggregationssystem, das darauf ausgelegt ist, benutzerdefinierte Pipelines zu erstellen, die Informationen aus verschiedenen Webquellen abrufen, filtern und anreichern. Es nutzt Large Language Models, um die Informationsfilterung zu automatisieren und Inhalte zu bewerten, um Rauschen zu entfernen und hochwertige Geschichten hervorzuheben. Das System integriert das Model Context Protocol, um Pipeline-Stufen als Tools für externe KI-Assistenten bereitzustellen. Es verwendet einen einheitlichen Adapter, um verschiedene KI-Modellanbieter für konsistente Inhaltsbewertungs- und Zusammenfassungsaufgaben zu standardisieren. Die Pipeline aggregiert Daten aus RSS-Feeds, sozialen Plattformen, Finanz-Toolkits und Code-Repositories. Sie verwaltet Inhalte durch Deduplizierung, kontingentbasierte Kategoriefilterung und kontextuelle Anreicherung, bevor sie mehrsprachige Briefings per E-Mail, Webhooks oder statischer Website-Bereitstellung liefert. Workflows werden durch wiederkehrende Cloud-Automatisierung orchestriert, um die geplante Sammlung und Bereitstellung verarbeiteter Informationen zu verwalten.

SD Maid SE ist ein Dienstprogramm zur Speicheroptimierung und Systemwartung für Android. Es konzentriert sich auf die Rückgewinnung von Speicherplatz durch die Analyse der Speichernutzung und das Entfernen doppelter, verwaister oder ungenutzter Dateien. Das Projekt zeichnet sich durch die Nutzung von Accessibility-Services aus, um repetitive Geräteaufgaben und manuelle Dateiüberprüfungen durch die Simulation von Benutzerinteraktionen zu automatisieren. Es enthält zudem spezialisierte Tools zur Reduzierung der Dateigröße von Bildern und Videos durch Medienkompression. Das System bietet ein breites Spektrum an Fähigkeiten, einschließlich Anwendungs-Lebenszyklusverwaltung zum Einfrieren oder Entfernen von Software, Junk-Bereinigung für Anwendungscaches und die Erkennung redundanter Daten durch Signaturvergleich. Es nutzt zudem hierarchisches Speichermapping zur Visualisierung der Festplattennutzung und unterstützt geplante Wartungsroutinen zur Erhaltung der Gerätegesundheit.

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

go-fastdfs is a distributed file system and object storage server designed for building private cloud storage. It provides a FastDFS compatible storage implementation that manages clusters of storage nodes to handle large-scale file uploads and downloads. The system focuses on high availability through a decentralized architecture that automatically synchronizes data and repairs failures across multiple machines without a central coordinator. It specifically supports resumable file storage via HTTP, allowing large transfers to be paused and resumed from the last successful byte to handle netw

Papra is a self-hosted document management system designed for digital archiving, organization, and retrieval. It serves as a centralized platform for storing files with a focus on security, providing an encrypted file archive using AES-256-GCM and a programmatic interface for managing documents and metadata via a REST API, SDK, and command line tools. The system distinguishes itself through an automated document ingestion engine that imports files via email forwarding, monitored folders, and webhook listeners. It further enhances discoverability by acting as an OCR document indexer, extracti

go-containerregistry is a Go library and toolkit for interacting with OCI and Docker registries. It provides a programmatic implementation of the Open Container Initiative distribution specification to fetch, upload, and manage container images, manifests, and layers. The library functions as a container image manipulation tool and a multi-platform image index manager. It enables the resolution and management of manifest lists that target various hardware architectures and operating systems without requiring a local daemon. The toolkit covers a broad range of registry interactions, including

This project is an automated content automation pipeline and AI article generator. It uses large language models to research topics from diverse web sources and academic repositories to generate evidence-based text and accompanying AI imagery for digital publishing. The system features a centralized social media management dashboard used to coordinate posting schedules, tone, and account positioning across multiple platforms. It employs a vector-based deduplicator to identify and remove redundant stories from the pipeline and uses topic clustering to rank content based on relevance. The work

fclones is a command-line tool designed to locate identical files across a filesystem by comparing file sizes and cryptographic hashes. It functions as a parallel filesystem scanner and a deduplication utility that identifies duplicate files to reclaim disk space. The tool distinguishes itself through a persistent hash cache system that stores hashes and metadata on disk to accelerate repeated scans. It employs a multi-phase scanning process and device-aware parallel I/O, which adjusts thread pools based on whether the storage is an SSD or HDD to maximize throughput. Beyond discovery, the pr