awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
Back to internlm/opencompass

Projects sharing features with Opencompass

30 open-source projects similar to internlm/opencompass, ranked by shared indexed features. Tags may describe platforms or build tools rather than the same primary purpose. Check each project’s use case, license, and deployment requirements before treating it as a replacement.

  • open-compass/opencompassopen-compass avatar

    open-compass/opencompass

    7,572View on GitHub↗

    OpenCompass is an open-source framework for standardized benchmarking of large language models. It provides a configurable evaluation pipeline that supports both objective and subjective assessment, using a dual-engine architecture to handle closed-form answer comparison and open-ended response rating. The framework is designed as a modular platform where datasets, models, and metrics are composed through declarative YAML configuration files. The framework distinguishes itself through its extensible model integration layer, which supports custom models, HuggingFace models, and third-party API

    Pythonbenchmarkchatgptevaluation
    View on GitHub↗7,572
  • oumi-ai/oumioumi-ai avatar

    oumi-ai/oumi

    8,858View on GitHub↗

    Oumi is a comprehensive large language model development platform designed for synthesizing data, fine-tuning models, and running performance evaluations. It serves as a unified environment for the entire model lifecycle, encompassing a training and fine-tuning suite, an evaluation framework, and tools for synthetic data generation and model distillation. The platform is distinguished by its iterative, failure-driven synthesis approach, which analyzes model weaknesses during evaluation to generate targeted training data. It utilizes an LLM-based judge framework to programmatically score respo

    Pythondpoevaluationfine-tuning
    View on GitHub↗8,858
  • openai/evalsopenai avatar

    openai/evals

    18,702View on GitHub↗

    Evals is a framework designed for automating, managing, and executing repeatable benchmarking suites to analyze the quality and performance of language models. It provides a platform for running standardized tests to measure model accuracy and track behavioral changes over time. The system distinguishes itself through a modular architecture that uses a standardized adapter layer to normalize inputs and outputs, allowing different models to be swapped and tested interchangeably. It supports the creation of custom benchmarks using proprietary data, enabling quality assurance on sensitive tasks

    Python
    View on GitHub↗18,702

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Find more with AI search
  • ibm/mcp-context-forgeIBM avatar

    IBM/mcp-context-forge

    3,310View on GitHub↗

    mcp-context-forge is a Model Context Protocol federation gateway that unifies diverse AI tool servers and APIs into a single consistent interface for discovery and execution. It acts as a centralized proxy that aggregates multiple servers and APIs, allowing AI agents to access and invoke a unified set of tools, prompts, and resources. The project distinguishes itself through a multi-protocol translation bridge that converts communication between standard I/O, SSE, gRPC, and REST to enable interoperability between disparate tool servers. It includes a comprehensive LLM evaluation framework for

    Pythonagentsaiapi-gateway
    View on GitHub↗3,310
  • agenta-ai/agentaAgenta-AI avatar

    Agenta-AI/agenta

    4,890View on GitHub↗

    Agenta is a Prompt Ops lifecycle manager and prompt management platform that decouples prompt engineering from application code. It serves as a centralized system for developing, versioning, and deploying prompt templates and model configurations across different environments. The platform functions as an AI agent orchestrator with a visual interface for building agent workflows and connecting models to external tools. It further acts as an evaluation framework and observability tool, utilizing OpenTelemetry to capture execution traces, monitor latency, and track token costs. The system cove

    TypeScriptagentsevaluationllm-as-a-judge
    View on GitHub↗4,890
  • open-compass/vlmevalkitopen-compass avatar

    open-compass/VLMEvalKit

    3,824View on GitHub↗

    VLMEvalKit is a vision-language model evaluation framework and inference engine designed to run standardized benchmarks and measure model accuracy across diverse visual datasets. It serves as a multimodal model benchmark and performance toolkit for calculating metrics and comparing model responses. The toolkit includes a specialized visual reasoning evaluator that uses adversarial samples to distinguish actual image understanding from reliance on language patterns. It also provides capabilities for image generation evaluation, testing a model's ability to create or modify visuals based on tex

    Pythonchatgptclaudeclip
    View on GitHub↗3,824
  • sgl-project/sglangsgl-project avatar

    sgl-project/sglang

    38,561View on GitHub↗

    Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

    Pythonattentionblackwellcuda
    View on GitHub↗38,561
  • arize-ai/phoenixArize-ai avatar

    Arize-ai/phoenix

    12,193View on GitHub↗

    Arize Phoenix is an LLM observability platform and evaluation framework designed to capture execution traces and monitor large language model applications. It serves as a prompt management system for versioning and testing templates, and as a self-hosted AI operations infrastructure for managing telemetry and experiments. The platform differentiates itself through a specialized embedding visualization tool used to detect data drift and optimize vector search. It provides a comprehensive evaluation suite that utilizes judge-based evaluators and ground-truth datasets to score model outputs, and

    Pythonagentsai-monitoringai-observability
    View on GitHub↗12,193
  • typpo/promptfootyppo avatar

    typpo/promptfoo

    22,295View on GitHub↗

    promptfoo is an evaluation framework for measuring the performance of large language model prompts, agents, and retrieval augmented generation pipelines. It provides a suite of tools for conducting comparative benchmarking and executing automated quality and security regressions. The system features a benchmarking suite for running identical prompts across different model providers to compare output quality side-by-side. It also includes a dedicated red teaming tool for identifying security vulnerabilities and prompt injection risks through automated penetration testing. The framework suppor

    TypeScript
    View on GitHub↗22,295
  • mlflow/mlflowmlflow avatar

    mlflow/mlflow

    26,554View on GitHub↗
    Pythonagentopsagentsai
    View on GitHub↗26,554
  • verazuo/jailbreak_llmsverazuo avatar

    verazuo/jailbreak_llms

    3,563View on GitHub↗

    This project is a comprehensive ecosystem of frameworks, toolkits, and datasets designed to evaluate model vulnerabilities and analyze jailbreak patterns. It serves as an adversarial testing framework and research toolkit for measuring the effectiveness of safety guardrails in large language models. The system includes a library of real-world prompt injection datasets harvested from social media to study bypass strategies. It provides specialized tools for semantic attack analysis and prompt visualization, allowing for the mapping of relationships between adversarial prompts to discover commo

    Jupyter Notebookchatgptjailbreakjailbreaking
    View on GitHub↗3,563
  • llm-attacks/llm-attacksllm-attacks avatar

    llm-attacks/llm-attacks

    4,509View on GitHub↗

    This repository provides tools and methodologies for studying adversarial attacks on large language models. It focuses on understanding how carefully crafted inputs can manipulate or bypass the safety mechanisms of LLMs, enabling researchers to probe model vulnerabilities and improve their robustness. The project covers techniques for generating adversarial prompts, evaluating model responses under attack conditions, and analyzing the effectiveness of different attack strategies.

    Python
    View on GitHub↗4,509
  • open-edge-platform/anomalibopen-edge-platform avatar

    open-edge-platform/anomalib

    5,871View on GitHub↗

    Anomalib is a PyTorch-based library for visual anomaly detection, offering a modular framework, a comprehensive model zoo, and a benchmarking suite designed for industrial defect detection. It provides a wide range of algorithms—including generative, discriminative, teacher-student, and vision-language approaches—that support unsupervised, few-shot, and zero-shot settings. The library enables deployment through model export to ONNX and OpenVINO for edge devices, and includes a no-code web application for training and inference. It also features a command-line interface for orchestrating multi

    Pythonanomaly-detectionanomaly-localizationanomaly-segmentation
    View on GitHub↗5,871
  • giskard-ai/giskardGiskard-AI avatar

    Giskard-AI/giskard

    5,434View on GitHub↗

    Giskard is an evaluation framework, testing library, and quality monitoring system for large language models and AI agents. It serves as a toolkit for quantifying model performance and reliability, providing specialized capabilities for validating retrieval-augmented generation pipelines. The project distinguishes itself through an automated red teaming tool and security scanner designed to identify vulnerabilities, prompt injections, and safety risks. It utilizes adversarial probing and synthetic edge case generation to quantify model robustness and detect information disclosure. The platfo

    Python
    View on GitHub↗5,434
  • comet-ml/comet-llmcomet-ml avatar

    comet-ml/comet-llm

    19,673View on GitHub↗

    Comet LLM is an observability platform and evaluation framework designed for large language model applications and agentic workflows. It functions as a system for tracing, monitoring, and debugging execution flows while providing tools for prompt optimization and the enforcement of AI safety guardrails. The platform distinguishes itself through a combination of model-based scoring and heuristic metrics to quantify output quality and detect hallucinations. It includes a dedicated prompt and agent optimizer with an interactive playground for refining templates and tool configurations. For retri

    Python
    View on GitHub↗19,673
  • swe-bench/swe-benchSWE-bench avatar

    SWE-bench/SWE-bench

    6,160View on GitHub↗

    SWE-bench is an automated evaluation framework that tests large language models on real-world software engineering tasks. It measures how effectively models can generate and apply code patches that resolve actual GitHub issues, using a standardized dataset and scoring system built around Docker-based patch verification against original project test suites. The framework provides curated benchmark datasets spanning comprehensive, fast, verified, multilingual, and multimodal evaluation splits, allowing targeted assessment of model capabilities across different programming languages and issue ty

    Pythonbenchmarklanguage-modelsoftware-engineering
    View on GitHub↗6,160
  • opennmt/opennmt-pyOpenNMT avatar

    OpenNMT/OpenNMT-py

    7,001View on GitHub↗

    OpenNMT-py is a PyTorch neural machine translation framework used for training and deploying neural machine translation and large language models. It functions as a distributed model training system, an inference engine, and a toolkit for fine-tuning large language models. The framework distinguishes itself with a dedicated toolkit for adapting large language models through low-rank adaptation, quantization, and instruction tuning. It also includes a neural machine translation server that allows trained models to be hosted and exposed via REST API endpoints. The project covers a broad range

    Python
    View on GitHub↗7,001
  • nvidia/isaac-gr00tNVIDIA avatar

    NVIDIA/Isaac-GR00T

    6,222View on GitHub↗
    Jupyter Notebook
    View on GitHub↗6,222
  • mosaicml/llm-foundrymosaicml avatar

    mosaicml/llm-foundry

    4,415View on GitHub↗

    llm-foundry is a training framework for large language models, providing a system for foundation model pre-training and supervised fine-tuning. It includes a distributed trainer for scaling workloads across multiple nodes and GPUs, a dataset streaming pipeline for loading data from cloud storage, and a parameter-efficient fine-tuning implementation. The framework distinguishes itself through its use of parameter sharding and high-throughput data streaming to maintain stability during large-scale training. It incorporates low-rank adaptation to reduce computational costs and uses eight-bit flo

    Pythondeep-learningllmneural-networks
    View on GitHub↗4,415
  • dusty-nv/jetson-inferencedusty-nv avatar

    dusty-nv/jetson-inference

    8,734View on GitHub↗

    jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

    C++caffecomputer-visiondeep-learning
    View on GitHub↗8,734
  • open-mmlab/mmdetectionopen-mmlab avatar

    open-mmlab/mmdetection

    33,026View on GitHub↗

    This project is a modular research toolkit designed for developing, training, and evaluating deep learning models for object detection, segmentation, and video instance tracking. It provides a flexible training engine that manages complex neural network execution, including distributed training, custom lifecycle hooks, and weight optimization. The framework is built around a hierarchical configuration system that allows users to define architectures, data pipelines, and training hyperparameters through composable, inheritable files. The project distinguishes itself through its highly modular

    Pythoncascade-rcnnconvnextdetr
    View on GitHub↗33,026
  • wandb/clientwandb avatar

    wandb/client

    11,128View on GitHub↗

    This project is a collection of utilities designed for machine learning experiment tracking, data versioning, and the observability of large language model applications. It provides a client for recording hyperparameters and metrics during training to visualize performance trends and compare different model versions. The tool includes a model evaluation framework that uses custom scorers and automated judges to assess the quality of generated text outputs. It also provides observability tools to monitor and debug the execution flow and runtime behavior of language model applications. The sys

    Python
    View on GitHub↗11,128
  • evolvinglmms-lab/lmms-evalEvolvingLMMs-Lab avatar

    EvolvingLMMs-Lab/lmms-eval

    4,490View on GitHub↗

    lmms-eval is a benchmarking system and performance analysis suite designed to measure the capabilities of large multimodal models. It provides a framework for evaluating models across text, image, audio, and video datasets, serving as a multimodal dataset orchestrator and benchmarking tool to quantify accuracy and efficiency. The project distinguishes itself through a unified multimodal message protocol that structures diverse media inputs for consistent model consumption. It features specialized benchmarking for audio, video, visual, document, and spatial reasoning, alongside tools for model

    Pythonagiaudio-evaluationbenchmark
    View on GitHub↗4,490
  • lmnr-ai/lmnrlmnr-ai avatar

    lmnr-ai/lmnr

    2,608View on GitHub↗

    Lmnr is an LLM observability platform and evaluation framework designed for tracing, logging, and monitoring language model executions. It provides the tools necessary to debug agent behavior, analyze performance, and identify failure patterns in AI agents. The platform differentiates itself through a trace-to-dataset pipeline that converts production logs into labeled test sets for regression testing. It includes a prompt-variant replay engine to compare different prompts or models side-by-side and a state-cached debugging system to replay agent loops without restarting the process. The sys

    TypeScriptagentsaiai-observability
    View on GitHub↗2,608
  • evidentlyai/evidentlyevidentlyai avatar

    evidentlyai/evidently

    7,137View on GitHub↗

    Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of

    Jupyter Notebookdata-driftdata-qualitydata-science
    View on GitHub↗7,137
  • googlecloudplatform/generative-aiGoogleCloudPlatform avatar

    GoogleCloudPlatform/generative-ai

    17,851View on GitHub↗

    This project is a development platform for managing the lifecycle of generative artificial intelligence models. It provides a unified environment for accessing, fine-tuning, and deploying large language models, serving as an orchestrator that handles the integration of diverse models into custom applications. The platform distinguishes itself by offering a managed infrastructure for hosting and scaling models, which removes the requirement for manual server maintenance or configuration. It includes integrated tools for supervised fine-tuning and vector embedding optimization, allowing for the

    Jupyter Notebookagentsgcpgemini
    View on GitHub↗17,851
  • kiln-ai/kilnkiln-ai avatar

    kiln-ai/kiln

    4,910View on GitHub↗

    Kiln is an LLM development workbench and evaluation framework designed for designing, testing, and optimizing prompts and AI agents. It functions as a multi-agent orchestrator and a RAG optimization tool, providing a visual interface for the iterative development of AI systems. The project distinguishes itself through a comprehensive fine-tuning pipeline that supports zero-code model training and reasoning distillation. It enables the creation of hierarchical multi-agent systems where specialized actors coordinate via tool calling, and it implements a Model Context Protocol server to expose t

    Python
    View on GitHub↗4,910
  • infrasys-ai/aiinfraInfrasys-AI avatar

    Infrasys-AI/AIInfra

    8,556View on GitHub↗
    Jupyter Notebookaiinfraaisystem
    View on GitHub↗8,556
  • eventual-inc/daftEventual-Inc avatar

    Eventual-Inc/Daft

    5,832View on GitHub↗

    Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

    Rustai-engineeringai-pipelinearrow
    View on GitHub↗5,832
  • huggingface/open-r1huggingface avatar

    huggingface/open-r1

    26,326View on GitHub↗

    Open-r1 is a framework designed for the large-scale training, distillation, and optimization of language models focused on complex reasoning and programming tasks. It provides a comprehensive suite of tools for managing distributed training jobs across multi-node clusters, enabling the development of high-performance models through reinforcement learning and supervised fine-tuning. The project distinguishes itself by integrating secure, containerized code execution environments directly into the training and evaluation lifecycle. By allowing models to run and verify code snippets against test

    Python
    View on GitHub↗26,326