cyclotruc/gitingest
Gitingest
Gitingest is a tool for extracting, converting, and estimating the token size of codebases to facilitate ingestion by large language models. It transforms GitHub repositories and local directories into a single formatted text file that serves as a structured context window for model analysis.
The utility includes a codebase token estimator to calculate file structure and total token counts, helping to determine the scale of the extracted content. It supports both public and private repositories through token-based authentication and respects gitignore configurations to filter out irrelevant paths and assets.
The system provides capabilities for repository-to-text conversion, local directory ingestion, and codebase statistics analysis. It can export the resulting digests to text files or standard output streams.
Features
- Git Repository to Text Conversion - Transforms a complete codebase into a single formatted text file designed for LLM context windows.
- LLM Context Preparation - Converts repository data into a formatted text structure specifically prepared as context for LLM prompts.
- Code Repository to LLM-Ready Text Transformers - Converts GitHub repositories and local directories into structured, LLM-optimized text formats.
- Source Code Ingestors - Converts local folders of source code into a structured text format for machine processing or sharing.
- Codebase Text Serializers - Concatenates entire codebase contents into a single formatted text block optimized for LLM context windows.
- Repository Text Exporters - Transforms remote GitHub repositories into prompt-friendly text files while respecting access permissions and ignore rules.
- Size and Gitignore Filtering - Implements filtering based on .gitignore rules and file size to optimize codebase ingestion for LLMs.
- Repository Content Extractors - Fetches and aggregates repository files while filtering out gitignored content and private data.
- Unified Repository Ingestors - Handles both local filesystem directories and remote repository URLs through a unified ingestion interface.
- Heuristic Token Estimators - Calculates approximate token counts using rule-based heuristics to estimate model context usage.
- Codebase Analysis Tools - Extracts source files and directory structures to analyze project layout and calculate total token size.
- Codebase Size and Token Count Reporters - Computes and reports the file structure, total extraction size, and estimated token count of a codebase.
- Codebase Size and Token Count Reporters - Analyzes codebase scale by calculating file structure, extraction size, and estimated token counts.
- Private Repository Access - Provides secure authentication to retrieve content from private version control repositories using tokens.
- Token-Based Authentication - Uses personal access tokens to securely authenticate and request data from private version control repositories.
- Web Scraping and Crawling - Converts Git repositories into text for LLM analysis.
