Open-source alternatives to Textdistance

fastText is a library and framework for word embedding generation, text vectorization, and supervised text classification. It provides tools to transform raw text into fixed-length vector representations and to train models that assign category labels to sentences or documents. The system utilizes subword-based vectorization and character n-gram embeddings, allowing it to generate meaningful vectors for words that were not present during training. To manage resource usage, it includes a quantized language model implementation that employs product quantization and dimensionality reduction to d

PyText is an extensible PyTorch-based framework for building, training, and deploying custom natural language processing models, including text classifiers, sequence taggers, and intent-slot predictors. It provides a modular toolkit that allows developers to assemble these models using pluggable registries for model architectures, data formats, and tensorizers, all configurable through YAML files without requiring code changes. The framework distinguishes itself through its comprehensive support for the full NLP model lifecycle, from training to production inference. It includes pre-built neu

:book: A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.

This project is a Unicode text repair tool and mojibake correction library designed to fix encoding glitches and restore original characters from mangled strings. It functions as a text encoding detector and a Unicode normalization tool to resolve issues where text has been incorrectly decoded. The library specializes in reversing multi-layered encoding errors and repairing complex mojibake patterns. It includes capabilities for detecting lossy encoding sequences, guessing byte encodings, and decoding non-standard UTF-8 variants. The toolset covers a broad range of text cleaning and normaliz

Natural language detection library for Go

Flair is a natural language processing framework for training and applying models for sequence labeling and text classification. It provides a system for generating word embeddings and identifying semantic entities within text. The framework includes a dedicated system for zero and few-shot learning, enabling text classification and entity extraction using minimal training examples by leveraging pre-trained knowledge. Its capabilities cover named entity recognition, sentiment analysis, and the training of specialized models using custom datasets. It also includes tooling for the visual highl

Gibran is an Elixir natural language processor, and a port of WordsCounted.

Nanoid is a library for generating unique, fixed-length identifiers designed for distributed systems and database indexing. It produces compact, URL-safe strings by mapping random byte values to a custom character set, allowing for consistent memory allocation and predictable indexing performance across independent nodes without the need for central coordination. The library distinguishes itself by utilizing system-level, cryptographically secure entropy sources to ensure that every generated identifier is statistically unpredictable. This approach provides resistance against collision attack

ChatALL is a desktop application that functions as a multi-model chat client and aggregator for artificial intelligence services. It enables users to send a single prompt to multiple AI models simultaneously, allowing for the side-by-side comparison of generated responses within a unified interface. The application distinguishes itself through a local-first approach to data management, ensuring that all conversation logs and user configurations are stored directly on the user's device. This architecture supports privacy and offline access while providing a centralized system for managing and

Code and data associated with the AmbiEnt dataset in "We're Afraid Language Models Aren't Modeling Ambiguity" (Liu et al., 2023)

Word Stemming in Go

Simple sentiment analysis with Ruby

Open Academic Research on Improving LLaMA to SOTA LLM

DAAT-CWS

Text preprocessing package for use in NLP tasks https://pypi.org/project/textcl/

OpenHands is an autonomous AI software engineer and coding assistant designed to execute software engineering tasks by interacting directly with codebases and development environments. It functions as a platform for running AI agents that can write code and manage files to automate complex development workflows. The system distinguishes itself through a container-based execution environment that isolates agent actions within a sandboxed Linux environment. It employs an autonomous agent loop of observation, planning, and action, supported by a standardized communication protocol that allows it

AllenNLP is a PyTorch-based research library and deep learning language toolkit designed for developing and training neural network architectures for linguistic tasks. It provides a distributed training system that coordinates data and gradients across multiple GPUs and a framework for integrating pretrained transformer architectures. The system distinguishes itself with a dedicated algorithmic bias mitigation tool used to identify and reduce bias in linguistic model predictions. It also includes model influence analysis to interpret predictions by calculating the influence of specific traini

MultimodalC4 is a multimodal extension of c4 that interleaves millions of images with text.

This repository contains custom pipes and models related to using spaCy for scientific documents.

porter stemmer