Awesome GitHub RepositoriesConversation Structure Validation

Oumi is a comprehensive large language model development platform designed for synthesizing data, fine-tuning models, and running performance evaluations. It serves as a unified environment for the entire model lifecycle, encompassing a training and fine-tuning suite, an evaluation framework, and tools for synthetic data generation and model distillation. The platform is distinguished by its iterative, failure-driven synthesis approach, which analyzes model weaknesses during evaluation to generate targeted training data. It utilizes an LLM-based judge framework to programmatically score respo

Harmony 是一个 AI SDK，旨在对对话进行 Token 化、格式化推理布局、解析原始输出并定义工具调用模式。它提供了一个系统，用于将结构化对话和工具调用转换为大型语言模型推理和训练所需的 Token 序列。 该项目包括一个输出格式化程序，将推理链和多通道输出结构化为一致的布局，以防止 Token 丢失。它还具有一个响应解析器，将原始完成 Token 和实时流转换回结构化的消息对象和角色。 该 SDK 通过用于定义可调用函数和命名空间的框架来管理工具集成。它还提供了实时 Token 解析、模型行为配置和有状态对话序列化的功能。

This project is a training pipeline and framework for developing Chinese language models based on the Llama 2 architecture. It functions as a distributed GPU trainer and dataset preprocessing toolkit designed for both the initial pre-training of baseline models and subsequent supervised fine-tuning. The system distinguishes itself through a specialized workflow for Chinese text, incorporating a data curation pipeline that uses similarity hashing for deduplication and a tokenization process that converts raw text into memory-mapped binary files for efficient disk access. It implements a superv