raga-ai-hubRagaAI-Catalyst

View on GitHub↗

16,150 stars·3,591 forks·Python·Apache-2.0·2 viewscatalyst.raga.ai↗

RagaAI Catalyst

RagaAI-Catalyst is a suite of software implementation tools providing an SDK, dashboard, and platform for monitoring, debugging, red-teaming, and evaluating agentic AI workflows. It serves as an observability framework for tracing the execution paths of large language models and multi-agent systems.

The project distinguishes itself through a security suite for automated red-teaming and vulnerability scanning to detect biases, alongside a centralized prompt registry that decouples templates from application code. It further provides an evaluation platform that combines synthetic data generation with custom metric frameworks to quantify model accuracy and reliability.

The system covers broad operational domains including agent behavioral observability, prompt lifecycle management, and the application of output guardrails to block undesirable content. Its monitoring capabilities include trace-based execution graphing, timeline-based event sequencing, and diagnostic tools for analyzing multi-agent interaction flows.

The core functionality is delivered via a Python library for recording tool calls and decision-making processes.

Features

Agent Execution Tracing - Implements a system for tracing agent reasoning, tool calls, and decision-making processes to debug complex workflows.
Agent Observability - Provides a comprehensive framework for monitoring and tracing the decision-making processes of autonomous agents.
Agent Debugging Tools - Provides specialized tools for analyzing interaction timelines and execution graphs to debug agent logic.
Agent Tracing SDKs - Provides a Python library for recording tool calls and decision-making processes to debug agent behaviors.

Star history

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

LLM Observability - Serves as a complete framework for tracing and monitoring execution paths of LLMs and multi-agent workflows.

Model Evaluation Metrics - Provides frameworks for quantifying model accuracy and reliability using specific performance benchmarks.

Model Red-Teaming - Provides a security suite for automated red-teaming and vulnerability scanning to detect model biases.

Output Guardrails - Implements a validation layer to intercept and block harmful or undesirable model responses.

Prompt Management - Organizes and maintains a central library of prompts to ensure consistency across environments.

Prompt Registries - Provides a centralized system for defining and versioning prompt templates independently of application code.

AI Red Teaming - Scans models for vulnerabilities and biases using synthetic test cases and automated detectors.

Adversarial Red Teaming Toolkits - Runs automated adversarial test cases to detect biases and safety failures in AI models.

LLM Evaluation - Measures the accuracy and reliability of LLM outputs using specialized metrics and automated judges.

Synthetic Data Generators - Produces artificial datasets used to facilitate the testing and evaluation of AI applications.

Agentic Workflow Generators - Generates artificial datasets used for stress testing and evaluating complex agentic AI workflows.

Model Output Safeguarding - Implements output guardrails to block undesirable content and ensure model responses align with safety guidelines.

Trace-Based Flow Visualizers - Captures sequential tool calls and model interactions to visualize the logic flow as a directed graph.

Agent Interaction Timelines - Orders asynchronous agent interactions on a linear timeline to identify latency bottlenecks and race conditions.

Agent Interaction Dashboards - Provides a visual interface for monitoring and analyzing execution graphs in multi-agent systems.

Model Evaluation and Benchmarking - Platform for managing and optimizing LLM project performance.