Open-source alternatives to Bert

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

Fairseq is a deep learning research toolkit and sequence-to-sequence framework built on PyTorch. It provides a system for training and deploying models that map input sequences to output sequences, with a primary focus on neural machine translation and speech recognition. The toolkit allows for the generation of text sequences through search algorithms such as beam search and nucleus sampling. It includes capabilities for producing synthetic parallel training data by translating monolingual text using reverse sequence models. The framework supports large scale model training through multi-de

GLM-130B is a pre-trained foundation model and bilingual large language model designed for natural language processing tasks in both English and Chinese. It functions as an autoregressive language model and text generator capable of producing long-form content and predicting missing phrases. The model utilizes an autoregressive blank-filling architecture and a bidirectional dense transformer to process text. This approach allows the system to transition between understanding context through masked language modeling and generating sequential text using specific mask tokens. The project covers

LASER is a cross-lingual sentence embedding library and multilingual text encoder. It functions as a parallel text mining tool that maps sentences from multiple languages into a shared vector space for similarity and classification tasks. The system converts raw text into fixed-length embeddings, enabling the discovery of translation pairs by calculating the vector distance between sentences. This shared representation allows for cross-lingual document classification, where a model trained on one language can be used to categorize documents in another. The library includes a sentence-piece t

This project is an educational platform and research toolkit designed to teach deep learning through a combination of mathematical theory, visual diagrams, and executable code. It provides a comprehensive environment for building, training, and evaluating neural networks, grounding complex concepts in interactive computational notebooks that allow for hands-on experimentation. The framework distinguishes itself by interleaving theoretical foundations—including linear algebra, calculus, and probability—with practical implementations across multiple industry-standard libraries. It supports flex

This repository serves as an educational resource for learning the foundational architectures of natural language processing through concise code implementations. It provides a structured collection of deep learning models designed to process and understand human language, focusing on the core mechanics of neural network sequence modeling and text analysis. The project distinguishes itself by offering direct, hands-on implementations of complex architectures, including Transformers, attention mechanisms, and word embedding generation. By utilizing tensor-based computational graphs and gradien

This project is an implementation of the Transformer-XL language model, a neural network architecture designed for long-context language modeling. It provides frameworks for training and deploying models that capture long-term dependencies and relationships in text sequences that extend beyond a fixed context window. The implementation supports both PyTorch and TensorFlow, allowing for distributed training across multiple GPUs and host nodes. It employs a recurrent mechanism to maintain coherence in extended sequences, utilizing segment-level recurrence and state-based memory reuse. The code

ERNIE is a development toolkit for training, fine-tuning, and deploying large language models built on the PaddlePaddle deep learning platform. It provides a comprehensive suite of core components, including an inference server for vision and language models, a training and fine-tuning toolkit, and a framework for building retrieval-augmented generation systems using private knowledge bases. The project features multimodal AI models capable of reasoning across text, images, and video to perform complex visual understanding and information extraction. It distinguishes itself through specialize

ALBERT

Transformers is a comprehensive library for machine learning that provides a unified interface for training, fine-tuning, and deploying transformer-based models. It supports a wide range of tasks, including text classification, language modeling, question answering, and sequence-to-sequence translation, while offering specialized architectures for both text and vision processing. The framework includes tools for managing the entire model lifecycle, from data preprocessing and tokenization to distributed training and inference. The library features extensive support for model optimization and

bert4keras is a lightweight reimplementation of the BERT transformer architecture for the Keras deep learning framework. It serves as a natural language processing toolkit and transformer model library used for text classification, sequence labeling, and semantic embedding extraction. The framework includes a sequence-to-sequence model system for question answering and text generation, as well as a model inference server to deploy trained transformers as web APIs for real-time predictions. Capabilities cover a broad range of natural language understanding tasks, including reading comprehensi

This project is a transformer-based language model and autoregressive text generator designed to predict the next token in a sequence to produce human-like prose and synthetic text. It functions as a large language model that utilizes a transformer architecture to learn linguistic patterns from large datasets for unsupervised multitask learning. The repository provides a distribution of pre-trained weights, enabling natural language processing tasks without requiring additional training. This allows the model to perform zero-shot task generalization by applying learned patterns to new tasks.

Trax is a deep learning framework and hardware-agnostic tensor engine designed for designing and training neural networks. It serves as a research tool providing high-level combinators for composing complex architectures, alongside a dedicated library for building transformer models and a toolkit for reinforcement learning. The framework is distinguished by its support for reversible and sparse transformer architectures, which reduce memory and computational overhead. It enables a single set of model instructions to execute across different hardware backends without changing the underlying co

This project is a comprehensive Python toolkit designed for natural language processing, research, and education. It functions as a linguistic data processor that provides a standardized framework for managing, cleaning, and analyzing large collections of annotated text corpora and lexical resources. The library distinguishes itself through its integration of both symbolic and statistical methods, allowing users to perform complex tasks ranging from rule-based grammar parsing to machine learning-driven classification. It offers a modular pipeline for text processing, enabling the transformati

SentencePiece is a text segmentation engine and tokenization library designed for machine learning workflows. It provides a comprehensive toolkit for transforming raw text into subword units or numerical identifiers, enabling consistent data representation for neural network training and inference. The library supports the training of segmentation models from raw text, allowing for the creation of custom vocabularies tailored to specific domain requirements. The project distinguishes itself through its byte-level encoding and fallback mechanisms, which ensure that every input can be represent

Gensim is an unsupervised natural language processing toolkit designed for topic modeling, word embedding training, and the processing of large-scale text corpora. It provides a framework for discovering latent themes and semantic structures in text without the need for labeled data. The toolkit is distinguished by its ability to handle datasets that exceed system memory through iterator-based data streaming from disk. It also supports distributed model training, allowing complex modeling tasks to be executed across computer clusters. The library covers a broad range of analysis capabilities

This project is a neural machine translation system used to build models that automatically translate text from one language to another. It utilizes sequence-to-sequence modeling to transform variable-length input sequences into corresponding output sequences. The system implements bidirectional recurrent neural network encoding and attention mechanisms to capture contextual information and focus on specific parts of the source text during translation. To manage training and inference, it employs separate computational graphs and supports distributing model layers across multiple GPU devices.

This project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz

Flair is a transformer-based natural language processing framework used to build and train models for text classification and sequence tagging. It provides a specialized library for generating contextual text embeddings and performing linguistic analysis. The framework includes dedicated tools for named entity recognition, including the identification of specialized biomedical entities across multiple languages. It further supports entity linking to map identified text mentions to unique entries within general or biomedical knowledge bases. The project covers a broad range of language analys

Entity-Relation-Extraction is a machine learning framework designed to identify entities and their logical connections within unstructured text. It functions as a pipeline that transforms raw documents into structured knowledge graphs by utilizing deep learning models and transformer architectures. The project distinguishes itself through a schema-driven approach, which maps extracted information to predefined relational templates to ensure output consistency. It employs a multi-stage process that combines sequence-labeling token classification with contextual encoding to delineate entity bou

DrQA is an open-domain question answering system that retrieves relevant documents from a large corpus and extracts specific answers to natural language questions. It is implemented as a neural network system that combines a document retrieval engine with a machine reading comprehension model. The system utilizes a two-stage pipeline architecture. A coarse-grained document retriever uses weighted word vectors to identify potential documents, while a fine-grained machine reading comprehension model identifies and extracts the exact text span containing the answer. The project also includes a

OpenNMT-py is a PyTorch neural machine translation framework used for training and deploying neural machine translation and large language models. It functions as a distributed model training system, an inference engine, and a toolkit for fine-tuning large language models. The framework distinguishes itself with a dedicated toolkit for adapting large language models through low-rank adaptation, quantization, and instruction tuning. It also includes a neural machine translation server that allows trained models to be hosted and exposed via REST API endpoints. The project covers a broad range

ChatterBot is a conversational AI framework and machine learning dialogue system used to build bots that generate automated responses. It functions as a multilingual natural language processing library and a vector-based knowledge base, utilizing logic adapters and statistical pattern matching to select the most confident response to user input. The system supports multilingual chatbot training and processing by using a design independent of specific linguistic rules. It employs semantic vector search to retrieve contextually accurate responses from a database of stored conversations and can

This project is a comprehensive toolkit for adapting large language models to the Chinese language, providing a specialized framework for fine-tuning, inference, and local deployment. It serves as a coordinated suite for language-specific adaptation, including tools for expanding tokenizers and implementing retrieval-augmented generation. The project distinguishes itself through a complete pipeline for model adaptation, featuring multilingual tokenizer expansion and a fine-tuning framework that supports instruction-based supervised training and adapter merging. It also includes a dedicated de

DeepPavlov is a conversational AI framework and deep learning NLP library designed for building end-to-end dialogue systems and chatbots. It functions as an NLP pipeline orchestrator that allows users to compose pre-trained models and text processing components into sequential data flows for complex linguistic tasks. The system is distinguished by its ability to act as a chatbot deployment server, exposing trained conversational models as web services via REST and Socket APIs. It utilizes JSON-based pipeline configurations and dynamic variable interpolation to decouple model logic from infras

fastText is a library and framework for word embedding generation, text vectorization, and supervised text classification. It provides tools to transform raw text into fixed-length vector representations and to train models that assign category labels to sentences or documents. The system utilizes subword-based vectorization and character n-gram embeddings, allowing it to generate meaningful vectors for words that were not present during training. To manage resource usage, it includes a quantized language model implementation that employs product quantization and dimensionality reduction to d