Awesome GitHub RepositoriesModel Response Validation Wrappers

Instructor is a schema enforcement and validation library designed to transform language model outputs into structured, type-safe data formats. It functions as a validation layer that uses Pydantic to ensure model responses conform to specific data models, acting as a tool for forcing large language models to return data in predefined schemas. The project differentiates itself through a recursive error-feedback loop that automatically retries requests when structural errors occur, passing validation failure messages back to the model to guide corrections. It also includes a streaming parser c

TypeChat is a schema enforcement library and framework for building natural language interfaces. It ensures that responses from large language models strictly adhere to predefined TypeScript type definitions, translating unstructured human language into predictable, structured data. The project functions as both a prompt generator and an output validator. It automatically creates model instructions by extracting requirements from type schemas to replace manual prompt engineering and verifies that model outputs match the required format. The system handles structured output generation and res

Garak is an AI model evaluation tool and vulnerability scanner designed for red teaming large language models and auditing the security of retrieval-augmented generation pipelines. It identifies behavioral weaknesses, such as jailbreaks, hallucinations, and data leakage, by simulating adversarial attacks and executing automated testing vectors. The framework utilizes an adaptive probing loop where prompts can react to previous model behavior and be modified in flight via middleware. To ensure consistent analysis, it employs a provider-agnostic interface to interact with various model APIs and

Poml is a prompt management framework and templating engine designed for authoring, versioning, and rendering structured prompts for large language models. It uses a semantic markup language to organize prompts into reusable templates, combining them with dynamic context and data to generate formatted inputs. The system distinguishes itself by decoupling core prompt logic from final presentation through a stylesheet-based approach. It provides a dedicated JSON schema output generator to enforce strict, machine-parsable model responses and a configuration interface for managing function tool s

FLAML is an automated machine learning framework, hyperparameter optimization tool, and large language model agent orchestrator. It provides a system for model selection and tuning across various learners and datasets, while also offering a toolkit for optimizing the inference parameters and fine-tuning settings of large language models. The project features a meta-learning tuning system that analyzes historical task data to generate data-dependent default configurations, accelerating model convergence. It further enables the design of collaborative multi-agent systems through conversational