Open-source alternatives to YuLan Mini

ChatGLM3 is an open-weights large language model designed for bilingual conversational interactions in English and Chinese. It functions as a tool-augmented system capable of calling external functions and executing internal code to resolve complex tasks. The model utilizes four-bit quantization to reduce memory requirements, enabling inference on consumer hardware and diverse processing units including GPUs and CPUs. It features an expanded context window for processing and summarizing long documents and includes a supervised fine-tuning pipeline for adapting the model to specialized domains

Skywork series models are pre-trained on 3.2TB of high-quality multilingual (mainly Chinese and English) and code data. We have open-sourced the model, training data, evaluation data, evaluation methods, etc.

This project is a transformer-based language model and autoregressive text generator designed to predict the next token in a sequence to produce human-like prose and synthetic text. It functions as a large language model that utilizes a transformer architecture to learn linguistic patterns from large datasets for unsupervised multitask learning. The repository provides a distribution of pre-trained weights, enabling natural language processing tasks without requiring additional training. This allows the model to perform zero-shot task generalization by applying learned patterns to new tasks.

This project is an educational implementation of a small-scale generative pre-trained transformer designed to teach the fundamentals of neural network architecture and training. It serves as a reference implementation and tutorial for constructing a text-generating neural network from scratch. The codebase demonstrates the mechanics of tokenization, self-attention, and the construction of a lightweight language model. It focuses on the step-by-step process of building a generative model to illustrate how large language models are constructed. The implementation covers transformer-based archi

A 13B large language model developed by Baichuan Intelligent Technology

🛝 Playground | 📄 Technical report | 💻 GitHub | 👀 Sign up for the API

Run inference on the latest MPT-30B model using your CPU. This inference code uses a ggml quantized model. To run the model we'll use a library called ctransformers that has bindings to ggml in python.

中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。

ChatYuan: Large Language Model for Dialogue in Chinese and English

RWKV-LM is a framework for training and deploying recurrent language models. It utilizes a linear-time recurrent architecture that enables text generation and sequence processing with constant memory and time complexity, avoiding the quadratic scaling of traditional attention caches. The project implements a parallelizable training mechanism that allows recurrent models to be trained using global operations while maintaining cache-free inference. It includes state-tuning capabilities to optimize the initial hidden state and utilizes adaptive probability-mass sampling to control token diversit

ChatRWKV is an open-source frontend and GPU-accelerated inference engine designed for interacting with RWKV recurrent neural network language models. It provides a self-hosted web chat interface and a specialized client for generating human-like text using a linear-complexity architecture. The project utilizes a GPU-accelerated backend that employs custom CUDA kernels and dynamic model format conversion to increase processing speed and reduce memory overhead. It manages conversation history through state-based context management, updating a fixed-size hidden state to maintain a constant memor

Dolly is an instruction-tuned large language model designed to follow complex natural language directions. It operates as a causal language model that predicts the next token in a sequence to generate coherent conversational responses and perform tasks such as brainstorming, classification, and question answering. The project focuses on the development of models using open datasets suitable for commercial application. It enables the creation of instruction-following models by utilizing curated collections of human-generated instruction-response pairs. The repository provides capabilities for

DeepSeek-LLM is a large language model and causal language model designed for natural language generation. It functions as a multi-lingual system capable of predicting the next token in a sequence to perform text completion and conversational generation. The model is specialized for logical reasoning, specifically as a code and math LLM. This enables it to perform complex problem solving, which includes generating executable code and solving mathematical equations through step-by-step analysis. The system's broader capabilities cover conversational AI, including the generation of chat comple

DeepSeek-V2 is a large language model designed for natural language processing and the analysis of long text sequences. It utilizes a mixture-of-experts architecture to balance high performance with inference efficiency. The model employs a sparse routing mechanism and shared expert neurons to capture common knowledge while maintaining specialization. It further reduces memory overhead and increases throughput through multi-head latent attention, group-query attention, and low-rank tensor compression. These capabilities enable the processing and retrieval of information from extensive token

This project is a training pipeline and framework for developing Chinese language models based on the Llama 2 architecture. It functions as a distributed GPU trainer and dataset preprocessing toolkit designed for both the initial pre-training of baseline models and subsequent supervised fine-tuning. The system distinguishes itself through a specialized workflow for Chinese text, incorporating a data curation pipeline that uses similarity hashing for deduplication and a tokenization process that converts raw text into memory-mapped binary files for efficient disk access. It implements a superv

An open-source educational chat model from ICALK, East China Normal University. 开源中英教育对话大模型。(通用基座模型，GPU部署，数据清理) 致敬: LLaMA, MOSS, BELLE, Ziya, vLLM

This repository is for EleutherAI's project Pythia which combines interpretability analysis and scaling laws to understand how knowledge develops and evolves during training in autoregressive transformers. For detailed info on the models, their training, and their properties, please see our…

Llama is a large language model runtime and inference engine designed to load and execute autoregressive transformer models. It enables the generation of natural language text completions from prompts using pretrained weights. The system features multi-GPU model parallelism, which distributes model weights and workloads across multiple graphics processors to support larger parameter counts. It also incorporates a content safety filter that uses classifiers to intercept and block unsafe inputs or outputs during the inference process. The project covers broad capabilities in distributed model

The official repo of Aquila2 series proposed by BAAI, including pretrained & chat large language models.

FlagAI is a distributed deep learning framework and platform designed for the end-to-end lifecycle of large-scale foundation models. It provides a toolkit for training, fine-tuning, and deploying large language models and multi-modal systems across multi-node computing clusters. The project features hardware-agnostic compute abstractions to ensure consistent execution across different accelerators. It includes a dedicated library for parameter-efficient fine-tuning, allowing large neural networks to be adapted to specific tasks with minimal parameter updates and reduced computational overhead

This repository serves as a comprehensive research platform and toolkit for advancing machine learning, quantum computing, and large-scale scientific data analysis. It provides foundational frameworks for developing complex algorithmic systems, offering the necessary infrastructure for distributed training, computational graph execution, and high-performance model development. The project distinguishes itself by integrating specialized research domains with robust, privacy-preserving methodologies. It supports diverse scientific discovery through tools for quantum simulation, physics-informed

Go to T5X ReadTheDocs Documentation Page.

This is a machine learning framework for treating diverse natural language processing tasks as a unified text-to-text problem. It provides a toolkit for pre-training and fine-tuning large-scale transformer models, utilizing a system where both inputs and outputs are formatted as raw text sequences. The framework is distinguished by its distributed training system, which uses mesh-based strategies to scale model weights and training batches across multiple TPU cores. It supports multi-task learning by combining diverse datasets into a single training stream using configurable mixture rates, al

Panda项目是于2023年5月启动的开源海外中文大语言模型项目，致力于大模型时代探索整个技术栈，旨在推动中文自然语言处理领域的创新和合作。