microsoft/LLMLingua
LLMLingua
LLMLingua is a prompt compression tool that reduces token count in prompts before they are sent to a large language model, cutting API costs and latency while preserving task performance. It operates as an extractive pipeline using a BERT-level Transformer encoder to classify each token for removal based on full bidirectional context from the prompt, retaining only key information and discarding non-essential tokens.
The tool is trained through a knowledge distillation process, where a compact compression model learns from an extractive dataset derived from a large language model's output to guide token removal decisions. This approach enables task-agnostic compression without requiring fine-tuning for out-of-domain inputs, and also helps mitigate the lost-in-the-middle effect in retrieval-augmented generation and chain-of-thought scenarios.
Beyond basic prompt compression, LLMLingua supports compressing long context prompts, segmenting conversation histories by topic for structured memory retrieval, and denoising retrieved memory units to improve accuracy. The library is designed to accelerate LLM inference by feeding compressed prompts instead of full-length inputs, and its capabilities extend to improving downstream task performance in RAG and CoT workflows.
Features
- Prompt Compressors - Ships a bidirectional Transformer encoder that classifies tokens for removal to compress prompts.
- Token Classification Compressors - Provides a BERT-level encoder that classifies tokens for removal using full bidirectional context.
- Extractive Prompt Compressors - Ships an extractive pipeline that selects key information from prompts and discards non-essential tokens.
- Extractive Distillation Pipelines - Trains a compact model on an extractive dataset derived from LLM output to guide token removal.
- Prompt-Based Accelerators - Accelerates LLM response times by compressing prompts before inference.
- Extractive Compression Distillers - Distills compression knowledge from LLMs by training a small model on an extractive dataset.
- LLM Cost Management - Reduces LLM inference costs by compressing prompts before API calls to cut token usage.
- Model Inference Accelerators - Accelerates model inference by feeding compressed prompts instead of full-length inputs.
- Redundancy Removers - Removes redundant tokens identified by a small language model to cut API costs and latency.
- Lost-in-the-Middle Mitigations - Provides prompt compression that reduces lost-in-the-middle effects to improve RAG and CoT task performance.
- Inference Acceleration - Accelerates LLM inference by feeding compressed prompts instead of full-length inputs.
- Bidirectional Context Compressors - Compresses prompts by classifying tokens for removal using a Transformer encoder with bidirectional context.
- Task-Agnostic Compressors - Compresses prompts task-agnostically using a distilled classifier trained on GPT-4 data.
- Context Compression - Compresses lengthy prompts or documents by retaining only key information to lower cost and latency.
- Lost-in-the-Middle Mitigators - Mitigates the lost-in-the-middle effect by compressing prompts to focus on essential context.
- Transformer-Based Reducers - Removes redundant tokens from prompts using a small Transformer encoder trained on GPT-4 data.
- Downstream Task Enhancers - Improves downstream task performance by compressing prompts to reduce lost-in-the-middle effects.
- Prompt Compression - Accelerates inference by compressing prompts and context.
- Prompt Engineering - Library for prompt compression to accelerate inference speeds.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
