Text metric calculator

This project is a comprehensive Python toolkit designed for natural language processing, research, and education. It functions as a linguistic data processor that provides a standardized framework for managing, cleaning, and analyzing large collections of annotated text corpora and lexical resources. The library distinguishes itself through its integration of both symbolic and statistical methods, allowing users to perform complex tasks ranging from rule-based grammar parsing to machine learning-driven classification. It offers a modular pipeline for text processing, enabling the transformati

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

BLEURT is an evaluation metric for Natural Language Generation. It takes a pair of sentences as input, a reference and a candidate, and it returns a score that indicates to what extent the candidate is fluent and conveys the meaning of the reference. It is comparable to sentence-BLEU, BERTscore,…

BERT Score is a text evaluation tool that assesses the quality of generated text by computing precision, recall, and F1 metrics between candidate and reference texts. It transforms text tokens into dense contextual vectors using pretrained transformer models, calculating token-level similarity matrices through pairwise cosine distances. The system computes scores by greedily matching tokens between sequences and supports multilingual assessment across dozens of different languages by utilizing language-specific or cross-lingual transformer backends. The library includes features for baseline

Machine learning evaluation metrics, implemented in Python, R, Haskell, and MATLAB / Octave

This is a Python fuzzy string matching library used for calculating string similarity and edit distances. It serves as a collection of string distance algorithms, a sequence alignment tool, and an approximate string search engine to measure text similarity. The library provides a wide array of metrics to quantify string closeness, including Levenshtein, Jaro-Winkler, Hamming, and Damerau-Levenshtein distances. It supports similarity analysis through longest common subsequence calculations, token-based comparisons, and weighted scoring to account for differences in content and word order. Bey

RapidFuzz is a C++ accelerated Python library providing high-performance string comparison and similarity calculations. It functions as a fuzzy string matching toolkit used to quantify the difference between text sequences through Levenshtein distance and other edit distance metrics. The library focuses on scalable approximate text matching, enabling the identification and ranking of similar strings within large datasets. It provides specialized utilities for finding the best matches in a collection and generating pairwise similarity matrices. The project covers a broad surface of text proce

Fuzzywuzzy is a Python library and text processing utility designed to calculate similarity scores between strings. It functions as a text similarity scoring engine and an approximate string matching tool used to identify the closest textual matches within a list of candidate strings. The library provides a suite of tools for measuring the degree of similarity between pieces of text, accounting for typos and formatting differences. These capabilities include extracting the best match from a candidate list and performing fuzzy string matching through various scoring methods. The toolset cover

This project is a library and command-line interface for local large language model inference. It enables the generation of text completions and chat responses from various model architectures. The project provides tools for weight quantization to reduce memory footprints and incorporates hardware acceleration through GPU offloading to increase computation speed. It also includes utilities for model evaluation by measuring perplexity on specific datasets. Capabilities cover the full inference lifecycle, including binary model loading, template-based prompt structuring, and session persistenc

Natural is a natural language processing library for Node.js that provides tools for text analysis, tokenization, and phonetic matching. It functions as a collection of specialized toolsets for word stemming, string similarity quantification, and pattern-based text classification. The library includes a phonetic sound analyzer that converts words into phonetic representations to identify matches based on sound rather than literal spelling. It also features a text classification engine that assigns categories to text inputs using trained models and pattern recognition. Additional capabilities

GPT2-Chinese is a Chinese language model implementation based on the GPT-2 architecture. It provides a causal language model trainer and a natural language generation tool designed for training and generating human-like Chinese text sequences. The system integrates a BERT tokenizer to process Chinese corpora into manageable units for machine learning. It enables the development of predictive text models that can generate specific patterns, such as news or poetry, through prompt-based text completion. The project covers a full workflow including text tokenization, model training using a trans

llama-rs is a local large language model inference engine implemented in Rust. It enables the execution of model computations on local hardware to generate text responses from user prompts. The project utilizes Rust-based tensor operations and direct-memory model mapping to handle high-performance linear algebra and efficient weight loading. It incorporates weight quantization to reduce the memory footprint of models by converting high-precision weights into smaller formats. The system includes a command-line interface for interactive chat sessions and one-off prompts, along with file-backed