5 repositorios
Awesome GitHub RepositoriesEvaluation Frameworks
Tools for measuring and scoring the accuracy of artificial intelligence agents against expected outputs.
Distinguishing note: Focuses specifically on automated accuracy scoring for AI agents, distinct from general software testing.
Explore 5 awesome GitHub repositories matching artificial intelligence & ml · Evaluation Frameworks. Refine with filters or upvote what's useful.
Awesome Evaluation Frameworks GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- agno-agi/agno
agno-agi/agno
40,717Ver en GitHubAgno is an agent operating system designed to manage the lifecycle, tool execution, and persistent state of autonomous agents across distributed infrastructure. It provides a unified runtime environment that wraps diverse agent frameworks into a consistent, interoperable protocol, allowing developers to build and deploy complex multi-agent systems that coordinate tasks and delegate sub-processes. The platform distinguishes itself through a robust governance and orchestration layer that includes human-in-the-loop approval gates, role-based access control, and a centralized API gateway. It feat
AgentOS logs and tracks evaluation results in a database to monitor performance trends and access run history through an integrated management platform.
Pythonagentsaiai-agents - simstudioai/sim
simstudioai/sim
28,796Ver en GitHubThis project is an AI agent orchestration platform that provides a visual environment for building, testing, and deploying complex automation workflows. It functions as a low-code development interface where users can chain discrete functional blocks into dependency-aware pipelines to integrate artificial intelligence with external data and services. The platform supports the creation of intelligent conversational agents, automated business processes, and multi-service API orchestrations within a unified workspace. The platform distinguishes itself through its event-driven integration engine,
Defines custom scoring metrics and model selection to evaluate AI-generated content quality.
TypeScriptagent-workflowagentic-workflowagents - hannibal046/awesome-llm
Hannibal046/Awesome-LLM
26,933Ver en GitHubThis project serves as a comprehensive, static directory of external resources dedicated to the study and application of large language models. It functions as a centralized discovery point for developers and researchers, aggregating foundational academic papers, technical documentation, and specialized tools within a structured, version-controlled knowledge base. The repository distinguishes itself through a multi-level classification system that organizes diverse technical domains, ranging from model training frameworks and inference optimization to AI safety and hallucination detection. By
LLM Evaluation: — a named example documented in this learning resource.
- mlflow/mlflow
mlflow/mlflow
26,554Ver en GitHubConfigure the LLM used to power evaluation judges by specifying the provider and model name in the judge definition.
Pythonagentopsagentsai - karpathy/llm-council
karpathy/llm-council
14,761Ver en GitHubLLM Council is a framework for orchestrating multi-model workflows that generates consensus-based responses by querying multiple language models simultaneously. It functions as a multi-model orchestrator that distributes user prompts across various endpoints, aggregates the resulting outputs, and synthesizes them into a single, unified final answer through a designated chairman model. The system distinguishes itself by implementing an anonymized peer review loop, which masks model identities during the evaluation phase to ensure that critiques and rankings are based solely on output quality r
Provides an automated framework for auditing deliberation processes and ensuring objective decision-making.
Python
