THUDM/GLM-130B
GLM 130B
GLM-130B is a pre-trained foundation model and bilingual large language model designed for natural language processing tasks in both English and Chinese. It functions as an autoregressive language model and text generator capable of producing long-form content and predicting missing phrases.
The model utilizes an autoregressive blank-filling architecture and a bidirectional dense transformer to process text. This approach allows the system to transition between understanding context through masked language modeling and generating sequential text using specific mask tokens.
The project covers capabilities for bilingual text generation, high-performance model inference, and large language model evaluation. It supports hardware-specific quantization to reduce memory usage and increase inference speed, alongside a configuration-driven system for measuring performance across various datasets.
Features
- Bilingual Language Models - Functions as a large-scale model trained for high proficiency and natural dialogue in English and Chinese.
- Bilingual Text Generation - A system for producing long-form content and predicting missing phrases in English and Chinese.
- Long-Form Text Generation - Produces sequential, long-form text from left-to-right using generative mask triggers.
- Masked Language Modeling - Employs training techniques to predict randomly hidden tokens within a sequence to learn semantic relationships.
- Natural Language Processing - Performs comprehensive natural language processing tasks across both English and Chinese languages.
- Masked - Predicts missing words or phrases within a sentence by identifying and replacing mask tokens.
- Transformer Language Models - Implements a transformer-based language model for sequential text generation.
- Pre-trained Models - Serves as a large-scale pre-trained architecture ready for downstream fine-tuning and general-purpose text generation.
- Bidirectional Processing Architectures - Utilizes a bidirectional processing architecture to analyze text sequences in both directions for deeper contextual understanding.
- Bilingual Embeddings - Maps English and Chinese characters into a shared high-dimensional vector space for cross-lingual semantic understanding.
- High-Throughput Text Inference - Optimizes model execution for high-volume text generation with low computational latency.
- Inference Acceleration Techniques - Increases text generation speed through hardware-specific optimizations and quantization techniques.
- Open-Weights Models - Provides a pre-trained model with publicly available weights for deployment and custom NLP tasks.
- Precision Quantization - Employs hardware-specific quantization to reduce model weight precision, lowering memory usage and accelerating inference.
- Autoregressive Text Generation - Implements a text generation system that predicts tokens sequentially by feeding previous outputs back into the model.
- Generative Mask Tokens - Uses specific mask tokens to trigger the transition from blank-filling mode to sequential text generation.
- Decoder Models - Bilingual open-source pre-trained language model.
- Foundation Models - Large-scale bilingual pre-trained language model.
- Large Language Models - High-performance model optimized for limited compute environments.
- Open Source Models - Open bilingual pre-trained model with high parameter count.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
