Awesome GitHub RepositoriesModel Benchmarks

JARVIS is a system for large language model task orchestration, deployment management, and automation benchmarking. It utilizes a task orchestrator to decompose complex requests into actionable steps and coordinates various expert models to synthesize final responses. The project includes an AI model deployment manager to handle the local deployment of expert models across different hardware scales. It further provides an AI workflow API consisting of web endpoints used to trigger automated task workflows and retrieve results from model selection stages. The framework incorporates an automat

This project is a comprehensive repository and curated index of resources, research papers, and development frameworks designed to support the construction and deployment of intelligent systems. It serves as a centralized knowledge base for developers seeking to navigate the technical landscape of artificial intelligence, ranging from foundational educational materials to specialized implementation guides. The repository distinguishes itself by providing structured directories for comparing generative artificial intelligence providers, including aggregated performance metrics, pricing data, a

llm-numbers is a set of calculation tools and benchmarks used to predict hardware requirements, token usage, and operational costs across various model tiers. It provides a cost and resource calculator based on formulas and benchmarks to estimate tokens, GPU memory, and operational expenses for large language models. The project includes a hardware requirement planner for calculating the VRAM and GPU memory needed to host models based on parameter counts. It also features a token estimator that converts word counts into token estimates to predict API billing and context window usage, alongsid

This project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co

This project is an LLM research orchestrator and autonomous AI agent framework designed to automate the scientific lifecycle. It functions as an end-to-end research pipeline and model training toolkit, managing everything from initial literature reviews and hypothesis testing to the final drafting of academic papers. The system is distinguished by its ability to convert unstructured academic PDFs into machine-executable knowledge layers, allowing agents to reproduce and extend research findings. It employs a two-loop orchestration architecture and a specialized research engineering skill libr

Awesome Coding Plan is a community-driven knowledge repository that provides a comparative analysis of subscription-based coding environments and artificial intelligence development tools. It functions as a tracker for developer tool costs, aggregating data on pricing structures, usage quotas, and token limits to assist in the selection of cloud-based coding services. The project utilizes a standardized framework to evaluate the performance and economic efficiency of various language models. By organizing technical metrics into a unified format, it allows for the objective assessment of proce