Llm

Acest proiect este o bibliotecă și o interfață de linie de comandă pentru inferența locală a modelelor de limbaj mari (LLM). Permite generarea de completări de text și răspunsuri de chat din diverse arhitecturi de modele.

Proiectul oferă instrumente pentru cuantificarea ponderilor pentru a reduce amprenta de memorie și încorporează accelerarea hardware prin offloading GPU pentru a crește viteza de calcul. Include, de asemenea, utilitare pentru evaluarea modelului prin măsurarea perplexității pe seturi de date specifice.

Capabilitățile acoperă întregul ciclu de viață al inferenței, inclusiv încărcarea binară a modelului, structurarea prompt-urilor bazată pe șabloane și persistența sesiunii pentru menținerea contextului conversațional. Suportă, de asemenea, orchestrarea sarcinilor, permițând secvențierea mai multor apeluri de model în pipeline-uri pentru operațiuni în mai mulți pași.

Features

Generative Text Inference - Processes natural language prompts to generate intelligent text completions via a library or CLI.
Hardware-Accelerated Inference - Utilizes GPU offloading and specialized hardware accelerators to increase the speed of model computations.
Local Inference Engines - Provides a runtime optimized for executing large language models on local consumer-grade hardware.
Model Loading - Implements mechanisms for efficiently loading model weights and configurations into memory for execution.
Weight Quantization - Compresses model weights into lower-precision formats to reduce memory footprint and enable consumer hardware execution.
Text Tokenizers - Segments raw text into discrete tokens as part of a high-performance inference loop.
Model Binary Formats - Parses binary file formats containing hyperparameters and vocabularies to initialize model architectures.
Interactive Chat REPLs - Provides a terminal-based read-evaluate-print loop for real-time interaction and testing with language models.
Prompt Templates - Manages text patterns and templates to guide model responses according to specific requirements.
LLM Workflow Orchestrations - Chains language model calls and processing steps into multi-step automated workflows.
Perplexity Calculators - Computes model perplexity based on cross-entropy loss to evaluate predictive performance.
Model Performance Evaluators - Quantifies the accuracy and reliability of models by computing perplexity on specific datasets.
Structural Prompt Design - Implements methodologies for organizing model instructions using templates to improve output predictability.
Session Persistence - Allows saving and loading the state of an interaction to maintain context across sessions.
Session State Persistence - Persists tool state and query history between sessions to maintain conversational context.
LLM Session State Management - Manages the persistence and initialization of model configurations and interaction state for user sessions.
LLM Task Chains - Enables sequencing multiple model calls into a pipeline to perform complex operations like multi-step summarization.
Inference Pipelines - Sequences multiple model calls into a linear workflow to perform complex multi-step operations.

Istoric stele

Graficul istoricului de stele pentru rustformers/llm

rustformersllmArchived

Name: rustformers/llm
Author: rustformers

View on GitHub

6,151 stele379 fork-uriRustApache-2.04 vizualizăridocs.rs/llm/latest/llm

Llm

Features

Generative Text Inference - Processes natural language prompts to generate intelligent text completions via a library or CLI.
Hardware-Accelerated Inference - Utilizes GPU offloading and specialized hardware accelerators to increase the speed of model computations.
Local Inference Engines - Provides a runtime optimized for executing large language models on local consumer-grade hardware.
Model Loading - Implements mechanisms for efficiently loading model weights and configurations into memory for execution.
Weight Quantization - Compresses model weights into lower-precision formats to reduce memory footprint and enable consumer hardware execution.
Text Tokenizers - Segments raw text into discrete tokens as part of a high-performance inference loop.
Model Binary Formats - Parses binary file formats containing hyperparameters and vocabularies to initialize model architectures.
Interactive Chat REPLs - Provides a terminal-based read-evaluate-print loop for real-time interaction and testing with language models.
Prompt Templates - Manages text patterns and templates to guide model responses according to specific requirements.
LLM Workflow Orchestrations - Chains language model calls and processing steps into multi-step automated workflows.
Perplexity Calculators - Computes model perplexity based on cross-entropy loss to evaluate predictive performance.
Model Performance Evaluators - Quantifies the accuracy and reliability of models by computing perplexity on specific datasets.
Structural Prompt Design - Implements methodologies for organizing model instructions using templates to improve output predictability.
Session Persistence - Allows saving and loading the state of an interaction to maintain context across sessions.
Session State Persistence - Persists tool state and query history between sessions to maintain conversational context.
LLM Session State Management - Manages the persistence and initialization of model configurations and interaction state for user sessions.
LLM Task Chains - Enables sequencing multiple model calls into a pipeline to perform complex operations like multi-step summarization.
Inference Pipelines - Sequences multiple model calls into a linear workflow to perform complex multi-step operations.

Alternative open-source pentru Llm

Proiecte open-source similare, clasificate după numărul de funcționalități comune cu Llm.

pytorch/executorch
pytorch/executorch
4,296Vezi pe GitHub
ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,
Pythondeep-learningembeddedgpu
Vezi pe GitHub4,296
pytorch/serve
pytorch/serve
4,354Vezi pe GitHub
This project is a PyTorch model serving framework designed to deploy and scale machine learning models in production via scalable network endpoints. It functions as a high-performance inference server, optimizer, and model lifecycle manager that handles model loading, request batching, and hardware acceleration. The system distinguishes itself through advanced orchestration and optimization capabilities, such as chaining multiple models into sequential workflows using execution graphs and employing dynamic batching to improve throughput and latency. It provides specialized support for generat
Java
Vezi pe GitHub4,354
vladmandic/sdnext
vladmandic/sdnext
7,139Vezi pe GitHub
SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing images and videos using diffusion models. It functions as a comprehensive tool for diffusion model management and an automated image processing pipeline for bulk operations. The project is distinguished by its hardware-backend abstraction layer, which provides automatic detection and acceleration for NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and DirectML. It features a headless generative API and a programmatic command interface, allowing users to trigger tasks via REST API or CLI wi
Pythonai-artcaptiondiffusers
Vezi pe GitHub7,139
intel-analytics/ipex-llm
intel-analytics/ipex-llm
8,836Vezi pe GitHub
ipex-llm is an acceleration library and inference engine designed to optimize the execution and finetuning of large language models on Intel GPUs and NPUs. It provides a HuggingFace compatible model backend and a dedicated quantization toolkit for converting model weights into low-bit precision formats. The project facilitates distributed inference by splitting large model workloads across multiple accelerators using pipeline and tensor parallelism. It enables the deployment of models on Intel Arc, Flex, and Max GPUs to increase throughput and reduce latency. The library covers a broad range
Python
Vezi pe GitHub8,836

Vezi toate cele 30 alternative pentru Llm

Întrebări frecvente

Ce face rustformers/llm?

Care sunt principalele funcționalități ale rustformers/llm?

Principalele funcționalități ale rustformers/llm sunt: Generative Text Inference, Hardware-Accelerated Inference, Local Inference Engines, Model Loading, Weight Quantization, Text Tokenizers, Model Binary Formats, Interactive Chat REPLs.

Care sunt câteva alternative open-source pentru rustformers/llm?

Alternativele open-source pentru rustformers/llm includ: pytorch/executorch — ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It… pytorch/serve — This project is a PyTorch model serving framework designed to deploy and scale machine learning models in production… vladmandic/sdnext — SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing… setzer22/llama-rs — llama-rs is a local large language model inference engine implemented in Rust. It enables the execution of model… intel-analytics/ipex-llm — ipex-llm is an acceleration library and inference engine designed to optimize the execution and finetuning of large… nunchaku-ai/comfyui-nunchaku — ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized…