Awesome GitHub RepositoriesPrompt Optimization Frameworks

This project provides a framework for managing multi-agent systems, designed to automate complex software development, infrastructure, and business workflows. It functions as a multi-agent workflow orchestrator that routes tasks to domain-specific workers while maintaining state persistence and infrastructure automation. By leveraging large language models, the system decomposes high-level objectives into actionable plans, ensuring that complex operations are executed with consistency and reliability. The framework distinguishes itself through its hierarchical agent registry and policy-driven

Comet LLM is an observability platform and evaluation framework designed for large language model applications and agentic workflows. It functions as a system for tracing, monitoring, and debugging execution flows while providing tools for prompt optimization and the enforcement of AI safety guardrails. The platform distinguishes itself through a combination of model-based scoring and heuristic metrics to quantify output quality and detect hallucinations. It includes a dedicated prompt and agent optimizer with an interactive playground for refining templates and tool configurations. For retri

Ragas is an evaluation framework designed to measure the performance of retrieval-augmented generation pipelines and autonomous agent workflows. It provides a comprehensive suite of tools for benchmarking system outputs, utilizing language models as automated judges to score performance against defined rubrics and reference data. By standardizing inputs, retrieved contexts, and generated responses into a unified schema, the project enables consistent analysis across complex AI applications. The framework distinguishes itself through its ability to generate synthetic test datasets from existin

PromptWizard 是一个自动化框架，专为大型语言模型指令的系统化优化而设计。它提供了一个结构化流水线，使用迭代反馈循环来评估、批评和优化自然语言提示，确保在各种生成任务中保持一致的性能。 该系统通过自反射优化脱颖而出，模型被要求根据自动化性能指标重写自己的指令。它通过集成逻辑推理链和专家角色进一步提高提示质量，同时策划少样本示例以引导模型输出模式。 除了基础优化外，该框架还包括用于生成合成数据以解决特定性能差距并扩展训练数据集的实用程序。用户可以通过可配置的超参数（包括变异轮次、批次大小和迭代次数的设置）控制进化过程的强度和范围。

This repository is a collection of specialized toolsets and libraries for large language model prompt engineering and security testing. It provides a library of advanced templates and frameworks designed to optimize the quality and specificity of model responses. The project includes resources for red teaming and security research, featuring a repository of prompts designed to bypass safety filters and operational constraints. It also provides techniques for system prompt extraction to reveal the internal instructions and configurations of AI personas. The collection covers a broader surface

TextGrad is a differentiable text optimization library and framework designed for simulated language model backpropagation. It functions as a textual gradient engine that treats language model feedback as gradients to iteratively refine prompts and unstructured text variables. The system utilizes a computation graph to trace errors from a defined loss function back to input text, allowing it to determine specific improvements. It differentiates itself by implementing natural-language backpropagation and gradient aggregation, which merges multiple pieces of textual critique into consolidated i

Automatic Prompt Engineer is a framework designed to automate the generation, refinement, and performance measurement of language model instructions. It functions as a systematic tool for optimizing prompt phrasing by iteratively testing candidate instructions against specific input and output datasets to maximize task accuracy. The system distinguishes itself through an evaluation-driven approach that uses automated feedback loops to score prompt variations. By employing template-based input structuring, it ensures consistent testing environments where candidate instructions are measured aga