Open-source alternatives to LMaaS Papers

gpt-neox is a distributed training system and framework for building large-scale autoregressive language models. It implements the transformer architecture and provides a toolkit for training models with billions of parameters by distributing weights across compute clusters. The framework distinguishes itself through extensive support for distributed model parallelism, including pipeline and sequence parallelism, to overcome single-device memory limits. It further supports sparse model architectures using a mixture of experts system with Sinkhorn-based routing. The project covers a broad ran

LAnguage Model Analysis

PyTorch original implementation of Cross-lingual Language Model Pretraining.

ColossalAI is a distributed deep learning framework designed for training and deploying massive artificial intelligence models across clusters of hardware accelerators. It functions as a parallel computing engine that partitions model workloads and data across multiple processors to maximize memory efficiency and throughput. The platform distinguishes itself through a comprehensive suite of parallelization strategies, including multi-dimensional tensor parallelism and pipeline-based model parallelism, which segment neural network layers and stages across devices. To support large-scale genera

This project is a PyTorch transformer model library and pre-trained model framework. It serves as a deep learning model hub and multimodal inference engine, providing a centralized system for loading, executing, and fine-tuning state-of-the-art model checkpoints. The library focuses on multimodal machine learning, enabling predictions across text, vision, and audio data. It provides specialized capabilities for model framework interoperability, allowing the conversion of weights and definitions between different deep learning libraries. The platform covers the full model lifecycle, including

Transformers is a comprehensive library for machine learning that provides a unified interface for training, fine-tuning, and deploying transformer-based models. It supports a wide range of tasks, including text classification, language modeling, question answering, and sequence-to-sequence translation, while offering specialized architectures for both text and vision processing. The framework includes tools for managing the entire model lifecycle, from data preprocessing and tokenization to distributed training and inference. The library features extensive support for model optimization and

GPT2-Chinese is a Chinese language model implementation based on the GPT-2 architecture. It provides a causal language model trainer and a natural language generation tool designed for training and generating human-like Chinese text sequences. The system integrates a BERT tokenizer to process Chinese corpora into manageable units for machine learning. It enables the development of predictive text models that can generate specific patterns, such as news or poetry, through prompt-based text completion. The project covers a full workflow including text tokenization, model training using a trans

bert中文分类实践

A List of Big Models

LARK is a development toolkit for training, fine-tuning, and deploying large language models and multimodal models based on PaddlePaddle. It functions as a comprehensive framework that includes an LLM training orchestrator, an inference server, and a multimodal model framework for processing text, image, and video inputs. The project features a retrieval-augmented generation system for building conversational applications that integrate web search and private knowledge bases. It provides specific capabilities for multimodal reasoning and complex logic, enabling the extraction of structured da

使用预训练语言模型ALBERT做中文NER

标注数据，可以说是AI模型训练里最艰巨的一项工作了。自然语言处理的数据标注更是需要投入大量人力。相对计算机视觉的图像标注，文本的标注通常没有准确的标准答案，对句子理解也是因人而异，让这项工作更是难上加难。 但是！谷歌最近发布的BERT大大的解决了这个问题！根据我们的实验，BERT在文本多分类的任务中，能在极小的数据下，带来显著的分类准确率提升。并且，实验主要对比的是仅仅5个月前发布的State of the art 语言模型迁移学习模型 - ULMFiT (https://arxiv.org/abs/1801.06146)， 结果有着明显的提升。

一行代码使用BERT生成句向量，BERT做文本分类、文本相似度计算

LASER multilingual sentence embeddings as a pip package

COLING 2022 CSL: A Large-scale Chinese Scientific Literature Dataset 中文科学文献数据集

Pre-trained Chinese ELECTRA（中文ELECTRA预训练模型）

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

Open Language Pre-trained Model Zoo

Kashgari is a production-level NLP Transfer learning framework built on top of tf.keras for text-labeling and text-classification, includes Word2Vec, BERT, and GPT2 Language Embedding.

中文 预训练 ELECTRA 模型: 基于对抗学习 pretrain Chinese Model

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo