6 Repos
Awesome GitHub RepositoriesModel Benchmarks
Comparative performance metrics, pricing data, and evaluation tools for generative artificial intelligence providers.
Distinct from Large Language Models: Distinct from Large Language Models: focuses on the comparative evaluation and benchmarking of providers rather than the models themselves.
Explore 6 awesome GitHub repositories matching artificial intelligence & ml · Model Benchmarks. Refine with filters or upvote what's useful.
Awesome Model Benchmarks GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- microsoft/jarvis
microsoft/JARVIS
24,854Auf GitHub ansehenJARVIS is a system for large language model task orchestration, deployment management, and automation benchmarking. It utilizes a task orchestrator to decompose complex requests into actionable steps and coordinates various expert models to synthesize final responses. The project includes an AI model deployment manager to handle the local deployment of expert models across different hardware scales. It further provides an AI workflow API consisting of web endpoints used to trigger automated task workflows and retrieve results from model selection stages. The framework incorporates an automat
Evaluates the capability of large language models to automate complex tasks using standardized benchmarking datasets.
Python - owainlewis/awesome-artificial-intelligence
owainlewis/awesome-artificial-intelligence
12,960Auf GitHub ansehenThis project is a comprehensive repository and curated index of resources, research papers, and development frameworks designed to support the construction and deployment of intelligent systems. It serves as a centralized knowledge base for developers seeking to navigate the technical landscape of artificial intelligence, ranging from foundational educational materials to specialized implementation guides. The repository distinguishes itself by providing structured directories for comparing generative artificial intelligence providers, including aggregated performance metrics, pricing data, a
Aggregates performance metrics, pricing data, and evaluation tools to facilitate objective comparison of generative artificial intelligence providers.
aiartificial-intelligencedeep-learning - ray-project/llm-numbers
ray-project/llm-numbers
4,310Auf GitHub ansehenllm-numbers ist eine Reihe von Berechnungstools und Benchmarks, die verwendet werden, um Hardwareanforderungen, Token-Nutzung und Betriebskosten über verschiedene Modellstufen hinweg vorherzusagen. Es bietet einen Kosten- und Ressourcenrechner basierend auf Formeln und Benchmarks, um Tokens, GPU-Speicher und Betriebsausgaben für Large Language Models zu schätzen. Das Projekt enthält einen Hardware-Anforderungsplaner zur Berechnung des VRAM- und GPU-Speichers, der zum Hosten von Modellen basierend auf Parameteranzahlen benötigt wird. Es bietet zudem einen Token-Schätzer, der Wortanzahlen in Token-Schätzungen umwandelt, um API-Abrechnungen und Kontextfensternutzung vorherzusagen, neben Preis-Benchmarks, die Kosten- und Durchsatz-Kompromisse zwischen verschiedenen Hosting-Methoden vergleichen. Das Toolset deckt KI-Modell-Benchmarking und Kostenprognosen, GPU-Ressourcenplanung und Leistungsanalyse zur Messung von Durchsatzgewinnen durch Batching ab. Es nutzt deterministische Formeln und statische Benchmark-Datensätze, um Parameter auf Speicher abzubilden und die Kosten-Nutzen-Verhältnisse zwischen Basismodellen und Fine-Tuning zu berechnen.
Provides comparative pricing and throughput benchmarks for different generative AI model tiers and hosting methods.
- openai/simple-evals
openai/simple-evals
4,354Auf GitHub ansehenThis project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co
Runs a suite of standardized benchmarks to measure language model accuracy on reasoning, math, and coding.
Python - orchestra-research/ai-research-skills
Orchestra-Research/AI-Research-SKILLs
3,641Auf GitHub ansehenThis project is an LLM research orchestrator and autonomous AI agent framework designed to automate the scientific lifecycle. It functions as an end-to-end research pipeline and model training toolkit, managing everything from initial literature reviews and hypothesis testing to the final drafting of academic papers. The system is distinguished by its ability to convert unstructured academic PDFs into machine-executable knowledge layers, allowing agents to reproduce and extend research findings. It employs a two-loop orchestration architecture and a specialized research engineering skill libr
Evaluates the ability of AI systems to autonomously design and analyze scientific experiments with rigor.
TeXaiai-researchclaude - mahonzhan/awesome-coding-plan
mahonzhan/awesome-coding-plan
1,641Auf GitHub ansehenAwesome Coding Plan is a community-driven knowledge repository that provides a comparative analysis of subscription-based coding environments and artificial intelligence development tools. It functions as a tracker for developer tool costs, aggregating data on pricing structures, usage quotas, and token limits to assist in the selection of cloud-based coding services. The project utilizes a standardized framework to evaluate the performance and economic efficiency of various language models. By organizing technical metrics into a unified format, it allows for the objective assessment of proce
Benchmarks processing speeds and token costs across different language models used for code generation.
