Open-source alternatives to Gpt Crawler

Reader is an AI data ingestion pipeline and web content parser designed to convert websites and documents into clean markdown for use with large language models. It functions as a headless browser content extractor and web-to-markdown converter, transforming URLs and PDF files into structured text formats while removing irrelevant web clutter. The system optimizes retrieval augmented generation by acting as a search optimizer that retrieves web results and applies re-ranking to improve context relevance. It further enhances content accessibility by using vision models to generate descriptive

This is a tool for searching, downloading, and archiving articles and engagement metadata from WeChat official accounts. It functions as a web-based content scraper and data exporter, allowing for the automated retrieval of social media content and the collection of performance metrics. The project distinguishes itself through a system that captures session credentials and authentication cookies from desktop clients via a local proxy to access private engagement data. It utilizes a concurrent proxy-pool fetching mechanism to download large volumes of content while avoiding rate limits, and it

This project is a high-performance headless browser engine designed for scalable web automation, data extraction, and AI agent integration. It provides a specialized environment that allows autonomous agents and testing frameworks to interact with web content through standardized remote control protocols. By executing pages in a lightweight, headless state, the engine minimizes resource consumption while maintaining the ability to perform complex navigation and dynamic content rendering. The platform distinguishes itself through deep integration with AI-centric communication layers and advanc

Undetected-chromedriver is a framework for automated browser navigation designed to bypass anti-bot security measures. It functions by patching browser drivers at the binary level to obscure automation signals, allowing scripts to interact with protected websites without being flagged or blocked by security services. The project distinguishes itself through its ability to maintain stealth during automated sessions, including those executed in headless mode. It achieves this by injecting custom configurations to mimic human user behavior and by hooking into low-level browser debugging protocol

Scylla is a system for managing HTTP proxy pools and automating web extraction. It provides a specialized data acquisition pipeline designed for gathering large-scale internet datasets for training and fine-tuning large language models. The project features a proxy rotation gateway that assigns fresh proxy addresses to incoming requests to mask origin traffic and avoid IP blocking. It includes a proxy pool manager that handles the collection, functional validation, and orchestration of proxy servers, complemented by a web dashboard for monitoring the health and geographic distribution of the

DataHub is a metadata management platform designed to unify technical, operational, and business context across diverse data ecosystems. By utilizing a graph-based metadata model and an event-driven ingestion architecture, it creates a centralized source of truth that maps complex data relationships, lineage, and ownership. This foundational framework enables organizations to maintain a synchronized view of their data landscape, supporting both human-led discovery and automated data operations. The platform distinguishes itself through its focus on grounding artificial intelligence and autono

Crawl4AI is an AI-powered web crawling and data extraction engine designed to transform complex web content into structured formats. It functions as a headless browser orchestrator, enabling the navigation of dynamic websites, the execution of custom scripts, and the capture of visual assets like screenshots and PDFs. By integrating language models directly into the extraction workflow, the system converts raw HTML into clean, structured data or Markdown files optimized for downstream ingestion. The platform distinguishes itself through a distributed, self-hosted infrastructure that manages l

recon-ng is an open source intelligence reconnaissance framework designed to automate the collection and aggregation of public information. It is a modular intelligence tool that utilizes a system of pluggable modules to harvest target data, resolve DNS queries, and parse web content. The framework is built as an API-driven tool with a programmatic interface to integrate with other security workflows. It is provided as a containerized application, using Docker to ensure a consistent environment for running reconnaissance tasks and managing a persistent data store. Its capabilities cover exte

Dokploy is a self-hosted platform-as-a-service designed to simplify the deployment and management of containerized applications and databases. It provides a centralized control plane that decouples administrative management from application workloads, allowing users to oversee infrastructure across multiple server nodes through a unified web interface or a command-line tool. The platform distinguishes itself through an extensive library of pre-configured application templates, enabling the rapid deployment of databases, identity providers, and various productivity or development tools. It sup

This project is a browser rendering service and headless Chrome PDF generator built on Puppeteer. It functions as a backend tool for converting web pages and raw HTML content into PDF documents and screenshots. The service distinguishes itself through browser session control, allowing for the injection of session cookies and the configuration of navigation timeouts to handle authenticated pages. It also includes viewport-based layout scaling to adjust browser dimensions and device scale factors during the rendering process. The broader capability surface covers HTML content export and automa

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Kilocode is an autonomous engineering platform designed to orchestrate AI agents for complex software development tasks. It functions as a comprehensive system for automating coding, testing, and repository management by integrating directly with your codebase and terminal. The platform provides a unified gateway for model orchestration, allowing for the management of agentic workflows, event-driven automation, and persistent session state across distributed development environments. The platform distinguishes itself through its federated task management and policy-based access control, which

This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer. The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-eva

Kotaemon is an orchestration framework designed for building modular, agentic workflows that integrate document processing, retrieval-augmented generation, and multi-step reasoning. It provides a comprehensive platform for developing document-based question answering systems, allowing users to chain language models, prompt templates, and external tools into complex, automated pipelines. The system distinguishes itself through a highly modular architecture that emphasizes component-based composition and schema-driven data exchange. It supports autonomous agents capable of decomposing complex q

memU is a long-term memory system for AI agents that provides a persistent knowledge base. It extracts facts and preferences from conversations into structured memories, organizing this information through a hierarchical knowledge base based on a file-system architecture of nested categories and linked resources. The system includes a multimodal data ingestion pipeline that converts audio, video, and images into standardized natural language for storage in large language model contexts. It also features a model provider abstraction layer, offering a unified interface to use interchangeable la

This project is an LLM knowledge base builder and personal knowledge management tool. It is a desktop application designed to transform diverse documents into a persistent, interlinked wiki through LLM analysis and incremental ingestion. The system distinguishes itself with a knowledge graph visualizer that uses community detection algorithms to map relationships between concepts and identify topical clusters. It features a hybrid retrieval system that combines keyword matching, vector embeddings, and graph relevance to locate information. The platform covers a wide range of capabilities inc

Screenpipe is a local screen and audio recorder that captures and indexes digital activity to create a searchable archive of computer usage. It functions as an AI context engine, providing a local database of visual and auditory history to ground large language models. The system serves as a Model Context Protocol server, delivering screen history and meeting transcriptions to external AI assistants. It utilizes an OCR screen search tool to extract text from visual data and a speech-to-text transcription tool for identifying speakers in system and microphone audio. The software includes capa

This project is a structured educational resource and technical guide for designing and implementing autonomous systems using large language models. It provides a comprehensive curriculum and code samples focused on agentic design patterns, autonomous development, and the creation of systems capable of planning and executing multi-step tasks. The resource details the implementation of agentic retrieval-augmented generation, where models autonomously plan and refine data searches. It covers a wide array of orchestrators and design patterns, including metacognitive reflection for self-correctin

OpenViking is a multi-tenant context server and knowledge base administration system designed to provide AI agents with persistent long-term memory. It enables the indexing of diverse documents and codebases to support retrieval-augmented generation, allowing agents to recall past interactions, user preferences, and learned experiences across sessions. The project is distinguished by its use of a URI-based virtual filesystem to organize memories, resources, and skills. It implements a tiered context loading system that balances retrieval precision with token budgets by structuring data into a

This project is an open-source intelligence reconnaissance framework and recursive attack surface mapper. It functions as a containerized security scanner designed to map public-facing infrastructure, perform subdomain enumeration, and automate the gathering of open-source intelligence. The system employs a recursive discovery engine to iteratively explore target infrastructure, utilizing a plugin-based module architecture to extend scanning capabilities. It integrates third-party APIs for data enrichment and applies YARA rules across discovered assets to identify specific vulnerability patte

jsdom is a Node.js DOM implementation that functions as a headless browser emulator and virtual browser environment. It provides a pure JavaScript implementation of web standards, acting as a web standards polyfill that simulates the window and document objects within a non-browser runtime. The project implements W3C and WHATWG specifications to provide a programmatic environment for parsing HTML and manipulating content. It serves as an HTML parser and serializer, allowing for the transformation of HTML strings into document structures and the export of those structures back into text. The

PhantomJS is a scriptable, headless browser engine based on WebKit that provides a programmatic interface for automating web page interactions. It operates without a graphical user interface, allowing for the execution of JavaScript to navigate pages, manipulate the document object model, and perform functional testing of web applications. The tool distinguishes itself by providing low-level control over the browser rendering lifecycle and network stack. It enables real-time interception and modification of network traffic, alongside the ability to generate visual snapshots and document expor

X-ray is a headless browser web scraper and HTML content crawler designed to extract structured data from websites. It functions as a stream-based data scraper and structured data extractor, using selectors to retrieve text and attributes from HTML as nested objects or arrays. The project includes a request rate controller to manage network traffic through concurrency limits, throttles, and timeouts. It handles dynamic website scraping by rendering JavaScript via a headless browser and performs automated website crawling using breadth-first link following and pagination management. The syste

Tika is a content analysis toolkit and Java library designed for detecting and extracting metadata and text from thousands of different file types. It functions as a universal document text extractor and metadata extraction engine, converting complex files into plain text or XHTML. The system employs a specialized MIME type detector that identifies document formats using magic bytes and metadata to determine the correct parser. It serves as an OCR integration gateway, connecting to external text recognition tools to extract content from image files. The project covers a broad range of extrac

Goutte is a PHP web scraper and DOM crawler designed for extracting data from websites. It functions as an HTTP client wrapper that enables the retrieval of web pages and the parsing of HTML content. The project provides a web form automator to programmatically fill and submit HTML forms to remote servers. It also includes a mechanism for automated website crawling by following links to discover and archive web content. The system supports stateful session management to maintain cookies and headers across requests. It further covers HTML data extraction through DOM-based element selection an

pydoll is a Chrome DevTools Protocol automation library and headless browser controller used for web data extraction and parallel browser automation. It controls Chromium-based browsers via direct WebSocket connections, allowing it to manage isolated browser contexts and tabs while bypassing the overhead and detection associated with WebDriver. The project features an anti-bot evasion framework that mimics natural human behavior, including mouse movements generated via Bezier curves and variable typing patterns. It provides specialized stealth capabilities to bypass behavioral analysis and au

html-to-markdown is a tool that transforms HTML content and websites into Markdown text. It functions as an HTML to Markdown converter and a command line processor for converting files or standard input streams into Markdown documents. The project is an extensible conversion engine that allows for custom element rendering and the registration of plugins to modify the transformation process. It includes a content filter that uses CSS selectors to include or exclude specific HTML elements during conversion. The system covers broad capability areas including link resolution for relative URLs, r