Awesome GitHub RepositoriesRecord Transformers

Fluentd is a unified logging layer and distributed event router that collects, parses, and routes log data from diverse sources to various storage backends. It functions as a log forwarding agent and pipeline orchestrator, transforming raw unstructured log strings into formatted objects using structured log parsing. The project utilizes a plugin-based pipeline architecture to route data through independent input, filter, and output stages. It differentiates itself through tag-based event routing, which uses regular expression patterns to direct specific data streams to their intended destinat

fp-ts is a TypeScript library that brings pure functional programming patterns to the language through algebraic data types, type class abstractions, and composable combinators. It provides foundational data types like Option for optional values, Either for typed error handling, and Task for lazy asynchronous computations, all designed to make invalid states unrepresentable and side effects explicit. The library is built on category theory concepts, offering type classes such as Functor, Applicative, Monad, Semigroup, and Monoid with lawful instances for common data structures. The library di

EmDash is an open-source content management system built on Astro that combines a visual admin panel with a plugin-driven architecture and server-side rendering. It provides a complete content management system with structured content modeling, a rich text editor using Portable Text format, and a TypeScript API for type-safe content queries. The system supports authentication through passkeys, OAuth 2.1, and external providers, with role-based access control and fine-grained permission scopes. What distinguishes EmDash is its plugin development framework, which supports both native plugins ru

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

Acest proiect este un framework de detecție a obiectelor TensorFlow conceput pentru antrenarea și implementarea modelelor Single Shot MultiBox Detector. Acesta oferă un toolkit de antrenare a rețelelor neuronale pentru implementarea arhitecturii SSD pentru a obține localizarea obiectelor în imagini și videoclipuri în timp real. Framework-ul include un pipeline de date dedicat pentru transformarea seturilor de date de detecție a obiectelor în formate de înregistrare binară pentru a crește viteza și performanța antrenării. De asemenea, dispune de utilitare pentru convertirea ponderilor modelului între diferite formate de checkpoint pentru a facilita reutilizarea rețelelor pre-antrenate. Sistemul acoperă o gamă largă de capabilități, inclusiv fine-tuning-ul modelului pe seturi de date personalizate, antrenarea detecției obiectelor și evaluarea acurateței prin măsurarea metricilor de precizie și recall.

Acest proiect este o implementare a arhitecturii modelului lingvistic ALBERT, oferind un framework pentru antrenarea și evaluarea clasificatorilor de text și a modelelor de similaritate bazate pe transformer. Acesta include în mod specific active pre-antrenate și instrumente optimizate pentru generarea de embedding-uri semantice și reprezentări ale textului chinezesc. Framework-ul se distinge prin instrumente pentru conversia checkpoint-urilor modelelor lingvistice grele în formate ușoare pentru a permite inferența cu latență scăzută pe dispozitive mobile. Utilizează tehnici specifice de reducere a ponderilor, inclusiv partajarea cross-parameter și parametrizarea embedding-urilor factorizate, pentru a menține performanța cu o amprentă de memorie mai mică. Sistemul acoperă un pipeline complet pentru procesarea limbajului natural, de la normalizarea textului brut și tokenizarea subword până la pre-antrenarea auto-supervizată folosind masked language modeling. Oferă capacități pentru adaptarea sarcinilor ulterioare, permițând modelelor pre-antrenate să fie reglate fin pentru analiza similarității textului și clasificare supervizată. Proiectul include utilitare pentru conversia datelor de înregistrare binară și transformarea formatului modelului pentru a asigura compatibilitatea între diferite platforme de machine learning.

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for

Erigon is an Ethereum execution client and blockchain archive node designed to maintain full or archive copies of the blockchain. It functions as a Proof-of-Stake validator, an Ethereum RPC server, and a network validator operation tool, providing the core infrastructure to validate blocks and secure the chain. The project distinguishes itself through high-performance storage and data distribution, utilizing a flat key-value state storage system instead of a Merkle Patricia Trie to increase read and write speeds. It employs BitTorrent-based data distribution for immutable historical blockchai