Awesome GitHub RepositoriesLinguistic Data Processors

spaCy is a Python natural language processing framework designed for industrial-scale text processing. It converts raw text into structured data for machine learning pipelines through a combination of statistical language model trainers, transformer-based text processors, and syntactic dependency parsers. The project enables the integration of pretrained transformer architectures to perform complex linguistic analysis and multi-task learning. It also provides a specialized system for neural named entity recognition to identify and categorize key entities within text. The framework covers a b

This project is a comprehensive Python toolkit designed for natural language processing, research, and education. It functions as a linguistic data processor that provides a standardized framework for managing, cleaning, and analyzing large collections of annotated text corpora and lexical resources. The library distinguishes itself through its integration of both symbolic and statistical methods, allowing users to perform complex tasks ranging from rule-based grammar parsing to machine learning-driven classification. It offers a modular pipeline for text processing, enabling the transformati

Chinese-xinhua is an open-source repository providing a comprehensive, machine-readable collection of Chinese linguistic data. It serves as a structured archive of dictionary entries, idioms, and phrases designed for programmatic access and integration into language processing applications. The project organizes complex linguistic information into consistent, schema-driven object structures that facilitate rapid lookups and data portability. By utilizing key-value indexing and structured text serialization, the dataset enables developers to implement advanced natural language search functiona

Stanza is a Python natural language processing library designed for tokenization, lemmatization, and dependency parsing across many human languages using neural models. It provides a neural processing pipeline that converts raw text into structured linguistic data objects, alongside a specialized analyzer for extracting medical insights from clinical and biomedical language. The project includes a wrapper that connects Python scripts to Java-based natural language processing tools and remote annotation servers. This enables a bridge for extracting linguistic annotations and analysis data from

This project serves as a comprehensive educational repository and technical reference collection, documenting a wide range of software engineering practices and modern development technologies. It provides a structured learning path for developers, curating tutorials and practical examples that cover the full lifecycle of application development, from initial project scaffolding to deployment and maintenance. The repository distinguishes itself by offering deep technical insights into complex architectural patterns, including actor-based concurrency models for managing parallel tasks and cont

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

Spark NLP es un kit de herramientas para el análisis de texto escalable y aprendizaje automático construido sobre el framework de computación distribuida Apache Spark. Proporciona un framework de aprendizaje automático multimodal y un sistema de tuberías distribuido para secuenciar anotadores para procesar datos lingüísticos a gran escala. La librería incluye un procesador de texto transformer para generar embeddings vectoriales contextuales y un motor de inferencia dedicado para gestionar grandes modelos de lenguaje. El proyecto se distingue por su capacidad para procesar tipos de datos heterogéneos, incluyendo texto, audio e imágenes, dentro de una arquitectura unificada de visión-lenguaje. Admite capacidades avanzadas de IA generativa como prompt engineering, extracción de entidades estructuradas con salida JSON restringida e inferencia local para eliminar la latencia de red. Además, proporciona herramientas para la traducción entre idiomas y la clasificación zero-shot a través de modalidades de texto e imagen. El framework cubre una amplia gama de capacidades, incluyendo el entrenamiento de modelos supervisados para el reconocimiento de entidades y el análisis de sentimientos, así como la respuesta a preguntas extractiva y el resumen de documentos. Integra soporte para bases de datos vectoriales para la búsqueda de similitud y ofrece infraestructura para la aceleración por GPU y la gestión del ciclo de vida del modelo a través de un registro centralizado. El kit de herramientas permite la distribución de modelos y tuberías personalizados a través de un repositorio público y admite el despliegue de modelos mediante APIs REST.