Llama Rs

llama-rs هو محرك استنتاج للنماذج اللغوية الكبيرة محلياً تم تنفيذه بلغة Rust. يتيح تنفيذ حسابات النموذج على الأجهزة المحلية لتوليد استجابات نصية من مطالبات المستخدم.

يستخدم المشروع عمليات الموتر القائمة على Rust وتعيين النموذج في الذاكرة المباشرة للتعامل مع الجبر الخطي عالي الأداء وتحميل الأوزان بكفاءة. يدمج تكميم الأوزان لتقليل أثر الذاكرة للنماذج عن طريق تحويل الأوزان عالية الدقة إلى تنسيقات أصغر.

يتضمن النظام واجهة سطر أوامر لجلسات الدردشة التفاعلية والمطالبات الفردية، إلى جانب استمرارية الجلسة المدعومة بالملفات لحفظ واستعادة سجلات المحادثة. كما يوفر أدوات لاسترجاع تكوينات المقسم (tokenizer) من المراكز البعيدة وأدوات لحساب درجات الحيرة لتقييم أداء النموذج.

Features

Inference Execution - Implements a local engine for running inference on large language models to generate text responses.
Command Line Inference Interfaces - Provides a terminal-based interface to interact with and manage model inference directly.
Memory-Mapped Weight Loaders - Uses memory-mapped weight loaders to efficiently map large model files into process memory.
Local Inference Engines - Acts as a runtime optimized for executing large language models on local consumer-grade hardware.
Model Quantization Frameworks - Uses a quantization framework to convert high-precision weights into smaller formats to reduce memory usage.
Weight Quantization - Compresses high-precision floating point weights into smaller integer-based formats to reduce memory footprint.
High-Performance Tensor Libraries - Utilizes high-performance tensor operations and linear algebra implemented in Rust for speed and safety.
Command-Line - Offers a terminal-based conversational interface for interacting with local AI models.
Conversation History Persistence - Saves and restores conversation histories to local storage to avoid reprocessing long prompts.
AI Companion Command-Line Interfaces - Provides a terminal-based interface for interactive chat sessions and one-off prompts.
AI Session State Management - Enables saving and loading chat histories to files or cache to reduce prompt processing time.
Language Models - A Rust-based port for running LLaMA-derived models.

سجل النجوم

setzer22llama-rsArchived

Name: setzer22/llama-rs
Author: setzer22

View on GitHub

6,150 نجوم377 تفرعاتRustApache-2.04 مشاهداتdocs.rs/llm/latest/llm

Llama Rs

Features

Inference Execution - Implements a local engine for running inference on large language models to generate text responses.
Command Line Inference Interfaces - Provides a terminal-based interface to interact with and manage model inference directly.
Memory-Mapped Weight Loaders - Uses memory-mapped weight loaders to efficiently map large model files into process memory.
Local Inference Engines - Acts as a runtime optimized for executing large language models on local consumer-grade hardware.
Model Quantization Frameworks - Uses a quantization framework to convert high-precision weights into smaller formats to reduce memory usage.
Weight Quantization - Compresses high-precision floating point weights into smaller integer-based formats to reduce memory footprint.
High-Performance Tensor Libraries - Utilizes high-performance tensor operations and linear algebra implemented in Rust for speed and safety.
Command-Line - Offers a terminal-based conversational interface for interacting with local AI models.
Conversation History Persistence - Saves and restores conversation histories to local storage to avoid reprocessing long prompts.
AI Companion Command-Line Interfaces - Provides a terminal-based interface for interactive chat sessions and one-off prompts.
AI Session State Management - Enables saving and loading chat histories to files or cache to reduce prompt processing time.
Language Models - A Rust-based port for running LLaMA-derived models.

بدائل مفتوحة المصدر لـ Llama Rs

مشاريع مفتوحة المصدر مشابهة، مرتبة حسب عدد الميزات المشتركة مع Llama Rs.

optimalscale/lmflow
OptimalScale/LMFlow
8,488عرض على GitHub
LMFlow is a comprehensive suite for large language model fine-tuning, context extension, multimodal processing, and inference execution. It provides a toolkit for updating model parameters through full tuning or memory-efficient adapter algorithms, alongside an inference engine for executing tuned models via command-line or web-based interfaces. The framework includes a dedicated alignment suite for supervised tuning and reward model training to refine model behavior. It features a context window extender to increase maximum input lengths and a multimodal framework for building chatbots that
Pythonchatgptdeep-learninginstruction-following
عرض على GitHub8,488
lm-sys/fastchat
lm-sys/FastChat
39,472عرض على GitHub
FastChat is a training and serving platform for large language models that provides an integrated toolkit for fine-tuning, hosting, and benchmarking chatbots. It functions as an inference server capable of hosting multiple models and exposing them via a standardized API for chat applications. The platform distinguishes itself through a distributed model controller that manages worker nodes and routes requests across a hardware-agnostic inference layer supporting various accelerators. It includes a dedicated evaluation framework for assessing model quality using automated judges, multi-turn di
Python
عرض على GitHub39,472
rustformers/llm
rustformers/llm
6,151عرض على GitHub
This project is a library and command-line interface for local large language model inference. It enables the generation of text completions and chat responses from various model architectures. The project provides tools for weight quantization to reduce memory footprints and incorporates hardware acceleration through GPU offloading to increase computation speed. It also includes utilities for model evaluation by measuring perplexity on specific datasets. Capabilities cover the full inference lifecycle, including binary model loading, template-based prompt structuring, and session persistenc
Rustaiggmlllm
عرض على GitHub6,151

الأسئلة الشائعة

ما هي وظيفة setzer22/llama-rs؟

ما هي الميزات الرئيسية لـ setzer22/llama-rs؟

الميزات الرئيسية لـ setzer22/llama-rs هي: Inference Execution, Command Line Inference Interfaces, Memory-Mapped Weight Loaders, Local Inference Engines, Model Quantization Frameworks, Weight Quantization, High-Performance Tensor Libraries, Command-Line.

ما هي البدائل مفتوحة المصدر لـ setzer22/llama-rs؟

تشمل البدائل مفتوحة المصدر لـ setzer22/llama-rs: optimalscale/lmflow — LMFlow is a comprehensive suite for large language model fine-tuning, context extension, multimodal processing, and… lm-sys/fastchat — FastChat is a training and serving platform for large language models that provides an integrated toolkit for… rustformers/llm — This project is a library and command-line interface for local large language model inference. It enables the… dusty-nv/jetson-inference — jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU… infrasys-ai/aisystem — AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip… meta-llama/llama-models — This project provides a foundational framework and reference implementation for executing causal language modeling and…

Llama Rs

Features

سجل النجوم

Llama Rs

Features

بدائل مفتوحة المصدر لـ Llama Rs

OptimalScale/LMFlow

lm-sys/FastChat

rustformers/llm

الأسئلة الشائعة

سجل النجوم

الأسئلة الشائعة

بدائل مفتوحة المصدر لـ Llama Rs

OptimalScale/LMFlow

lm-sys/FastChat

rustformers/llm

dusty-nv/jetson-inference