Projects sharing features with Stanza

This project is a comprehensive Python toolkit designed for natural language processing, research, and education. It functions as a linguistic data processor that provides a standardized framework for managing, cleaning, and analyzing large collections of annotated text corpora and lexical resources. The library distinguishes itself through its integration of both symbolic and statistical methods, allowing users to perform complex tasks ranging from rule-based grammar parsing to machine learning-driven classification. It offers a modular pipeline for text processing, enabling the transformati

nlp.js is a JavaScript natural language processing library and development framework used to build natural language understanding engines. It provides a toolkit for creating local machine learning models for intent classification and acts as a multilingual text processor that detects languages and normalizes text across various dialects. The framework distinguishes itself by supporting local execution on both servers and mobile devices, enabling chatbot functionality without an internet connection. It features a specialized system for conversational slot filling to collect mandatory informati

Flair is a natural language processing framework for training and applying models for sequence labeling and text classification. It provides a system for generating word embeddings and identifying semantic entities within text. The framework includes a dedicated system for zero and few-shot learning, enabling text classification and entity extraction using minimal training examples by leveraging pre-trained knowledge. Its capabilities cover named entity recognition, sentiment analysis, and the training of specialized models using custom datasets. It also includes tooling for the visual highl

pkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch

Transformers Tasks is a collection of toolkits and scripts dedicated to language model fine-tuning, natural language processing tasks, and transformer-based pipelines. The project functions as a natural language processing toolkit and transformer pipeline library, providing Python scripts and algorithms designed to adapt foundational language models and route text inputs through modular processing workflows. The repository covers supervised fine-tuning pipelines and reinforcement learning alignment procedures that optimize generative text outputs through reward modeling and policy gradient lo

Compromise is a natural language processing library and rule-based text parser designed to analyze unstructured text. It functions as a toolkit for identifying parts of speech, linguistic patterns, and semantic meaning, while providing specialized engines for named entity recognition and the parsing of temporal and numeric data. The project is distinguished by its linguistic morphological engine, which can conjugate verbs across different tenses and inflect nouns and adjectives. It further allows for linguistic model customization through a plugin system that enables the extension of lexicons

This project is an educational collection of tutorials and executable code notebooks focused on data science, machine learning, deep learning, and natural language processing concepts in Python. It provides instructional resources covering statistical analysis, linear algebra, artificial intelligence algorithms, and step-by-step guides for developers learning data science. The repository covers a broad spectrum of computational and statistical capabilities, including neural network construction, gradient-based optimization techniques, curve fitting, regression modeling, and collaborative filt

HanLP is a natural language processing library and deep learning framework specifically optimized for the Chinese language, while also functioning as a multilingual text processor. It serves as a toolkit for performing linguistic analysis, semantic understanding, and script conversion. The project distinguishes itself through a dedicated focus on Chinese linguistic structures, including a specialized script converter for transforming text between Simplified Chinese, Traditional Chinese, and Pinyin. It further supports domain-specific model training to improve the recognition of professional t

This repository serves as an educational resource for learning the foundational architectures of natural language processing through concise code implementations. It provides a structured collection of deep learning models designed to process and understand human language, focusing on the core mechanics of neural network sequence modeling and text analysis. The project distinguishes itself by offering direct, hands-on implementations of complex architectures, including Transformers, attention mechanisms, and word embedding generation. By utilizing tensor-based computational graphs and gradien

Knwl.js is a JavaScript named entity recognition library and rule-based text parser. It serves as an extensible information extraction tool designed to identify and pull structured entities, such as dates, times, and locations, from unstructured text strings. The library allows for the definition of specialized rules and custom plugins to identify and extract unique pieces of information. This extensibility enables the automation of information retrieval by converting human-readable text into structured formats for applications and databases. The system utilizes regular expression matching a

nlp-recipes is a collection of implementation guides and reference templates for applying natural language processing techniques to real-world tasks. It provides standardized workflows and code examples for developing NLP pipelines, from dataset preparation and model training to performance evaluation. The project focuses on the practical application of transformer-based models, offering patterns for fine-tuning pretrained architectures for tasks such as text classification, named entity recognition, and question answering. It also includes a toolkit for model interpretability, allowing users

Natural is a natural language processing library for Node.js that provides tools for text analysis, tokenization, and phonetic matching. It functions as a collection of specialized toolsets for word stemming, string similarity quantification, and pattern-based text classification. The library includes a phonetic sound analyzer that converts words into phonetic representations to identify matches based on sound rather than literal spelling. It also features a text classification engine that assigns categories to text inputs using trained models and pattern recognition. Additional capabilities

TextBlob is a natural language processing library that provides a unified interface for common linguistic tasks. It operates as a wrapper-based API, simplifying the use of complex processing libraries by delegating core operations to specialized external frameworks. The project features a pluggable processing pipeline that allows for the integration of custom logic and alternative language engines. It supports the extension of processing models through plugins to add specific language support or custom data processing. The library covers a broad range of linguistic capabilities, including se

spaCy is a Python natural language processing framework designed for industrial-scale text processing. It converts raw text into structured data for machine learning pipelines through a combination of statistical language model trainers, transformer-based text processors, and syntactic dependency parsers. The project enables the integration of pretrained transformer architectures to perform complex linguistic analysis and multi-task learning. It also provides a specialized system for neural named entity recognition to identify and categorize key entities within text. The framework covers a b

snips-nlu is a Python library and natural language understanding engine designed to convert unstructured text into structured data. It identifies user intents and extracts associated entities from natural language sentences to enable machine-readable command processing. The engine functions as a multilingual parser capable of processing text across multiple languages. It maps identified entities to canonical values or standardized ISO formats, such as timestamps, to ensure data consistency. The project covers intent classification and named entity recognition, utilizing sequence labeling and

This project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz

PyText is an extensible PyTorch-based framework for building, training, and deploying custom natural language processing models, including text classifiers, sequence taggers, and intent-slot predictors. It provides a modular toolkit that allows developers to assemble these models using pluggable registries for model architectures, data formats, and tensorizers, all configurable through YAML files without requiring code changes. The framework distinguishes itself through its comprehensive support for the full NLP model lifecycle, from training to production inference. It includes pre-built neu

DeepKE is a knowledge extraction toolkit and framework designed to transform unstructured text into structured knowledge graphs. It provides a pipeline for identifying and classifying named entities, semantic relations, and events, converting raw datasets into structured triples. The project utilizes large language models as tool callers through a standardized context protocol to drive automated data extraction processes. It supports schema-driven extraction across multiple domains and bilingual text, employing joint entity and relation extraction to identify components in a single structured

Spark NLP is a toolkit for scalable text analysis and machine learning built on the Apache Spark distributed computing framework. It provides a multimodal machine learning framework and a distributed pipeline system for sequencing annotators to process large-scale linguistic data. The library includes a transformer text processor for generating contextual vector embeddings and a dedicated inference engine for managing large language models. The project distinguishes itself through its ability to process heterogeneous data types, including text, audio, and images, within a unified vision-langu

NLP, before and after spaCy

This project is a natural language processing system designed for named entity recognition and text classification. It uses a machine learning approach to identify specific names and key information from raw text to organize unstructured content into a structured format. The system implements a multi-layer architecture that combines a pre-trained transformer for embeddings, bidirectional long short-term memory for sequence modeling, and a conditional random field for label transitions. It supports transfer learning through the fine-tuning of these models on task-specific datasets. The projec

YSDA course in Natural Language Processing

This project is a collection of scripts and workflows for training, fine-tuning, and deploying large language models using the Hugging Face Transformers toolkit. It functions as a distributed training framework, a library for natural language processing task implementations, and a system for building retrieval-augmented generation chatbots. The repository includes specialized tools for model optimization, such as a Bayesian hyperparameter optimizer for automatically tuning model settings. It provides implementations for scaling model training across multiple graphics processors using data par

This project is an educational course and learning curriculum for implementing and fine-tuning transformer models using the Hugging Face ecosystem. It serves as a structured guide and technical walkthrough for processing multimodal data, adapting pre-trained neural networks, and deploying models. The material includes a guide for managing, versioning, and distributing model weights and datasets through a centralized asset hub. It also provides a practical tutorial on adapting models to specific datasets using parameter-efficient methods and an implementation guide for solving natural language

PaddleNLP is a development library and toolkit for training, fine-tuning, and deploying large and small language models using the PaddlePaddle framework. It provides a comprehensive suite for the entire natural language processing lifecycle, from model development to high-performance inference. The project features a standardized model zoo for loading and managing pre-trained models and tokenizers through a unified interface. It distinguishes itself with a specialized model compression framework that reduces memory footprints via weight precision conversion and lossless size optimization, alo

This repository provides a collection of deep learning models and neural network architectures built for natural language processing tasks. It functions as a library of pre-trained models designed to process, analyze, and generate human language data using the TensorFlow framework. The project utilizes sequence-to-sequence modeling and layered neural architectures to handle variable-length language data. By employing static dataflow graphing and tensor-based representations, the models execute mathematical operations to transform input features into abstract linguistic meanings. Users can loa

This project is a collection of transformer natural language processing tutorial notebooks and educational resources. It provides a guide for using the Hugging Face Transformers library through interactive coding exercises and demonstrations. The repository contains ready-to-run Jupyter notebooks that provide practical examples for implementing transformer models. These resources demonstrate how to execute specific natural language processing workflows using pre-trained models. The notebooks cover a range of natural language processing tasks, including text classification, automatic text sum