Awesome GitHub RepositoriesPrompt Optimization Frameworks

This project provides a framework for managing multi-agent systems, designed to automate complex software development, infrastructure, and business workflows. It functions as a multi-agent workflow orchestrator that routes tasks to domain-specific workers while maintaining state persistence and infrastructure automation. By leveraging large language models, the system decomposes high-level objectives into actionable plans, ensuring that complex operations are executed with consistency and reliability. The framework distinguishes itself through its hierarchical agent registry and policy-driven

Comet LLM is an observability platform and evaluation framework designed for large language model applications and agentic workflows. It functions as a system for tracing, monitoring, and debugging execution flows while providing tools for prompt optimization and the enforcement of AI safety guardrails. The platform distinguishes itself through a combination of model-based scoring and heuristic metrics to quantify output quality and detect hallucinations. It includes a dedicated prompt and agent optimizer with an interactive playground for refining templates and tool configurations. For retri

Ragas is an evaluation framework designed to measure the performance of retrieval-augmented generation pipelines and autonomous agent workflows. It provides a comprehensive suite of tools for benchmarking system outputs, utilizing language models as automated judges to score performance against defined rubrics and reference data. By standardizing inputs, retrieved contexts, and generated responses into a unified schema, the project enables consistent analysis across complex AI applications. The framework distinguishes itself through its ability to generate synthetic test datasets from existin

PromptWizard is an automated prompt engineering framework designed to evolve natural language instructions for generative tasks. It functions as an in-context learning optimizer and synthetic data generator, using mutation rounds and performance metrics to iteratively refine large language model instructions. The system employs a self-reflective optimization loop that uses model-generated critiques to rewrite prompts. It distinguishes itself through the use of reasoning chain integration and persona-based prompting to steer the tone and professional quality of model responses. The framework

This repository is a collection of specialized toolsets and libraries for large language model prompt engineering and security testing. It provides a library of advanced templates and frameworks designed to optimize the quality and specificity of model responses. The project includes resources for red teaming and security research, featuring a repository of prompts designed to bypass safety filters and operational constraints. It also provides techniques for system prompt extraction to reveal the internal instructions and configurations of AI personas. The collection covers a broader surface

TextGrad is a differentiable text optimization library and framework designed for simulated language model backpropagation. It functions as a textual gradient engine that treats language model feedback as gradients to iteratively refine prompts and unstructured text variables. The system utilizes a computation graph to trace errors from a defined loss function back to input text, allowing it to determine specific improvements. It differentiates itself by implementing natural-language backpropagation and gradient aggregation, which merges multiple pieces of textual critique into consolidated i

Automatic Prompt Engineer es un framework diseñado para automatizar la generación, el refinamiento y la medición del rendimiento de las instrucciones de modelos de lenguaje. Funciona como una herramienta sistemática para optimizar la redacción de prompts probando iterativamente instrucciones candidatas contra conjuntos de datos de entrada y salida específicos para maximizar la precisión de la tarea. El sistema destaca por un enfoque impulsado por la evaluación que utiliza bucles de retroalimentación automatizados para puntuar variaciones de prompts. Al emplear la estructuración de entrada basada en plantillas, asegura entornos de prueba consistentes donde las instrucciones candidatas se miden contra métricas de rendimiento predefinidas. El framework incluye utilidades integradas para gestionar los recursos computacionales y financieros requeridos para las tareas de optimización. Proporciona funciones de pre-computación para estimar el uso de tokens y los costos antes de la ejecución, permitiendo el control presupuestario durante las pruebas a gran escala. Los usuarios pueden definir plantillas de evaluación y prompts personalizados para estandarizar cómo los modelos interactúan con los datos y cómo se puntúa el rendimiento a través de diferentes iteraciones.