Awesome GitHub RepositoriesText Processing Pipelines

spaCy is a Python natural language processing framework designed for industrial-scale text processing. It converts raw text into structured data for machine learning pipelines through a combination of statistical language model trainers, transformer-based text processors, and syntactic dependency parsers. The project enables the integration of pretrained transformer architectures to perform complex linguistic analysis and multi-task learning. It also provides a specialized system for neural named entity recognition to identify and categorize key entities within text. The framework covers a b

This project provides a framework for managing multi-agent systems, designed to automate complex software development, infrastructure, and business workflows. It functions as a multi-agent workflow orchestrator that routes tasks to domain-specific workers while maintaining state persistence and infrastructure automation. By leveraging large language models, the system decomposes high-level objectives into actionable plans, ensuring that complex operations are executed with consistency and reliability. The framework distinguishes itself through its hierarchical agent registry and policy-driven

Zincsearch is a high-performance, self-hosted full-text search engine and database written in Go. It provides a lightweight infrastructure for indexing and searching unstructured text data, specializing in log and event analysis through a schemaless indexing model. The system is designed as a resource-efficient alternative to heavier search infrastructure, featuring an API surface compatible with Elasticsearch for indexing and querying documents. It distinguishes itself by packaging the entire server and its built-in web search interface into a single statically linked binary. The engine cov

Transformers.js is a JavaScript library and web machine learning framework designed to run pretrained transformer models directly in the browser. It serves as a client-side inference engine and a wrapper for the ONNX Runtime, enabling the execution of multimodal AI tasks on user devices without the need for a backend server. The library distinguishes itself by providing a unified toolkit for processing text, image, and audio data locally. This architecture supports privacy-preserving model inference and reduces latency by performing all computations on the client's hardware. Its capabilities

Qwen3-Coder is a specialized large language model designed for software development, technical reasoning, and automated code synthesis. Built on transformer-based sequence modeling, it functions as a multilingual programming assistant capable of generating, completing, and debugging source code across more than one hundred programming languages. The model distinguishes itself through its capacity to process and maintain logical coherence across massive datasets, supporting context windows of up to one million tokens. This allows for repository-scale reasoning, enabling the model to analyze co

This project is a comprehensive Python toolkit designed for natural language processing, research, and education. It functions as a linguistic data processor that provides a standardized framework for managing, cleaning, and analyzing large collections of annotated text corpora and lexical resources. The library distinguishes itself through its integration of both symbolic and statistical methods, allowing users to perform complex tasks ranging from rule-based grammar parsing to machine learning-driven classification. It offers a modular pipeline for text processing, enabling the transformati

Flair is a transformer-based natural language processing framework used to build and train models for text classification and sequence tagging. It provides a specialized library for generating contextual text embeddings and performing linguistic analysis. The framework includes dedicated tools for named entity recognition, including the identification of specialized biomedical entities across multiple languages. It further supports entity linking to map identified text mentions to unique entries within general or biomedical knowledge bases. The project covers a broad range of language analys

Flexsearch is a full-text search library designed to execute complex indexing and retrieval operations directly within the web browser. By operating entirely on the client side, it enables rapid information lookup without requiring a backend server, ensuring that search functionality remains self-contained and private. The library distinguishes itself through a highly configurable text processing pipeline that allows developers to tailor tokenization and normalization to specific language requirements. To maintain interface responsiveness during large-scale queries, it utilizes background thr

TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on accelerated hardware. It functions as a framework that transforms machine learning models into efficient execution graphs, providing an engine to refine these models for specific hardware to maximize throughput and minimize latency during text generation. The project distinguishes itself through advanced execution strategies that manage the entire inference pipeline. It utilizes kernel-level fusion and static graph execution to optimize mathematical operations and computational f

Pipecat is a framework and software development kit for building real-time multimodal AI agents and speech-to-speech systems. It utilizes a frame-based data pipeline to route audio, video, and text through a modular sequence of processors, enabling the orchestration of low-latency conversational AI. The project is distinguished by its ability to coordinate complex multimodal services, including speech-to-text, language models, and text-to-speech, within a single pipeline. It features semantic voice activity detection for natural turn-taking, state-machine conversation flows for dialogue manag

LunaTranslator is a real-time translation tool designed for visual novels and games. It functions as a multi-engine translation hub and text extractor that captures dialogue via memory hooking or optical character recognition to convert it into a target language. The project distinguishes itself through specialized linguistic tools, including a Japanese text analyzer for sentence segmentation and phonetic readings. It also operates as a digital dictionary aggregator, querying multiple online and offline databases simultaneously to provide comprehensive vocabulary definitions for language lear

Airllm is a framework designed to execute and fine-tune large language models on consumer-grade hardware. By employing layer-wise model decomposition and memory-efficient loading techniques, the engine enables the operation of massive models that would otherwise exceed available system or video memory. The project distinguishes itself through a suite of optimization strategies that balance memory footprint with performance. It utilizes block-wise weight quantization and asynchronous layer prefetching to reduce resource consumption and hide data transfer latency. Additionally, the framework su

Bleve is a search indexing engine library written in Go, designed to provide full-text search and document retrieval capabilities for embedded application data. It functions as a framework for indexing structured or unstructured information, allowing developers to build searchable collections that support complex query logic and data analysis. The engine distinguishes itself through a pluggable analysis pipeline that normalizes text before indexing, alongside support for vector similarity search to identify semantically related content. It utilizes finite-state transducer automata for efficie

This project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz

CoreNLP est une bibliothèque Java de traitement du langage naturel conçue pour convertir du texte brut en données structurées. Elle utilise une suite d'annotateurs linguistiques pour analyser le texte via un pipeline, extrayant ainsi les structures grammaticales, les sentiments et les modèles linguistiques. Le projet inclut un moteur de résolution de coréférence qui lie les mentions multiples d'une même entité pour maintenir la cohérence contextuelle entre les documents. Il fournit également des outils de reconnaissance d'entités nommées pour catégoriser les personnes, les entreprises et les lieux, ainsi qu'un étiqueteur morphosyntaxique pour assigner des catégories grammaticales et des formes de base aux mots. La bibliothèque couvre en outre l'analyse de sentiment pour évaluer le ton émotionnel et prend en charge la sérialisation des données linguistiques traitées dans des formats standardisés pour le stockage ou l'échange externe.

Up is an interactive shell pipeline tool and Linux pipeline builder designed for prototyping text-processing sequences. It provides a terminal user interface for constructing chains of shell commands while displaying real-time data transformations. The tool allows for the iterative development of command sequences with an instant live preview of processing results. Once a sequence is finalized, it functions as a shell script generator that exports the completed pipeline into a reusable script file. The workspace includes capabilities for terminal data exploration and text processing workflow

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

Stanza is a Python natural language processing library designed for tokenization, lemmatization, and dependency parsing across many human languages using neural models. It provides a neural processing pipeline that converts raw text into structured linguistic data objects, alongside a specialized analyzer for extracting medical insights from clinical and biomedical language. The project includes a wrapper that connects Python scripts to Java-based natural language processing tools and remote annotation servers. This enables a bridge for extracting linguistic annotations and analysis data from

Translate-shell is a command-line translation tool and terminal dictionary client. It allows for the translation of words, phrases, and sentences between multiple languages and provides dictionary definition retrieval and language metadata display directly within the terminal. The tool functions as a shell-based text translator that can process input from standard streams, local files, or URLs. It includes text-to-speech capabilities to play audio pronunciations of source and translated text and can automatically detect the source language of a given string. The system supports interactive s