Awesome GitHub RepositoriesData Lakes

SeaweedFS is a distributed object store and high-performance file system designed to manage massive volumes of unstructured data. It utilizes a decoupled architecture that separates metadata management from raw data storage, allowing for independent scalability and the efficient handling of billions of files. By providing a POSIX-compliant interface, it enables applications to interact with a unified namespace while maintaining the performance characteristics of a distributed object store. The system distinguishes itself through a multi-region data fabric that supports active-active replicati

Hub is a multimodal AI data lake and vector database designed for storing and querying embeddings, text, audio, and images. It functions as a dataset version control system and a machine learning data streaming engine to support large-scale model training. The system utilizes a serverless PostgreSQL vector store to index high-dimensional embeddings for semantic search. It provides a visual interface for inspecting multimodal datasets and viewing annotations such as bounding boxes and masks. The platform handles cloud-agnostic storage synchronization and implements lazy, compressed data strea

DeepLake is AI data infrastructure consisting of a multimodal data lake, a hybrid search engine, and a serverless vector database. It provides a PostgreSQL-based AI data runtime that combines multimodal storage with streaming pipelines to load and shuffle datasets from cloud storage directly into deep learning training pipelines. The system utilizes lazy indexing to store and slice images, audio, and video without loading entire files into memory. It enables retrieval-augmented generation by persisting high-dimensional embeddings in a serverless vector store and implementing hybrid search tha

Scorecard ist ein Open-Source-Sicherheitsscanner und ein Tool zur Analyse der Software-Lieferkette, das den Sicherheitsstatus von Projekten durch die Berechnung von Risikometriken basierend auf Best Practices bewertet. Es fungiert als Dashboard für die Sicherheitsgesundheit und visualisiert Sicherheitslücken durch Scores und Badges, um Maintainer bei der Identifizierung von Schwachstellen zu unterstützen. Das Projekt bietet ein System zur Überwachung der Repository-Sicherheit durch einen GitHub-Action-Sicherheitsauditor, der Maintainer alarmiert, wenn Sicherheits-Scores sinken. Es bietet zudem einen Mechanismus für Anleitungen zur Behebung von Schwachstellen, der identifizierte Sicherheitslücken auf präskriptive Anweisungen zur Verbesserung der Entwicklungspraktiken abbildet. Das Tool deckt eine breite Oberfläche an Funktionen ab, einschließlich Open-Source-Sicherheitsaudits, CI/CD-Sicherheitsautomatisierung und der Analyse von Drittanbieter-Repositories zur Risikobewertung vor der Integration. Es unterstützt verschiedene Schnittstellen für die Interaktion, einschließlich eines Command-Line-Interfaces für Scans und eines REST-Interfaces zum Abrufen vorab berechneter Sicherheitsmetriken.

SlateDB is a cloud-native key-value store and distributed database engine that utilizes a log-structured merge-tree architecture. It serves as a transactional storage layer designed to persist data directly to cloud object storage. The engine differentiates itself by optimizing read performance for remote storage through the use of bloom filters and multi-level block caching. It employs a single-writer multi-reader model and provides the ability to create zero-copy clones via copy-on-write checkpointing. The system supports atomic transactions, range queries, and snapshot-based concurrency c