Projects sharing features with Crawlab

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Chalice is a framework for building and deploying serverless applications and REST APIs on AWS Lambda using Python. It functions as an infrastructure-as-code generator, mapping application logic and routing definitions directly to cloud compute resources while automating the provisioning and management of the underlying environment. The framework distinguishes itself by analyzing source code to automatically construct the minimum necessary security permissions, ensuring least-privilege access for all deployed functions. It supports modular development through blueprint-based organization and

Goutte is a PHP web scraper and DOM crawler designed for extracting data from websites. It functions as an HTTP client wrapper that enables the retrieval of web pages and the parsing of HTML content. The project provides a web form automator to programmatically fill and submit HTML forms to remote servers. It also includes a mechanism for automated website crawling by following links to discover and archive web content. The system supports stateful session management to maintain cookies and headers across requests. It further covers HTML data extraction through DOM-based element selection an

Katana is a web crawler and spider designed for security reconnaissance and web application mapping. It functions as a utility for identifying endpoints, forms, and API structures across web targets by combining standard HTTP request traversal with headless browser automation to render dynamic, JavaScript-heavy content. The tool distinguishes itself through its ability to maintain authenticated sessions and handle complex web interactions, such as automated form submission and captcha resolution. It provides granular control over the discovery process, allowing users to define specific crawl

Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a distributed spider engine and a headless browser renderer to execute JavaScript and extract content from dynamic web pages. The system includes a scalable data deduplicator to filter duplicate URLs and records during large-scale operations. A crawler monitoring system tracks the health of active scraping jobs and triggers alerts when system anomalies occur. The framework provides capabilities for task scheduling, web data extraction, and resilient workflows that allow crawling tas

Firecrawl is a headless browser automation tool and web crawling engine designed to extract structured data from the web. It functions as an API that transforms raw website content and documents into clean markdown and JSON formats to serve as context for large language models. The project distinguishes itself by using natural language prompts to translate human instructions into targeted data extraction tasks and browser actions. It can execute interactive page navigation, such as clicking and scrolling, and perform automated web research to retrieve structured data without manual interventi

Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

This project is a Python-based automation toolkit designed to manage programmatic authentication and session persistence across web services. It provides a framework for executing automated login sequences, including the handling of interactive security challenges such as QR code verification and captcha resolution. The toolkit distinguishes itself by simulating native mobile application environments, allowing for the execution of scripts that require specific device-level headers and behaviors. It also incorporates hook-based interception to monitor workflow states and manage exceptions duri

Colly is a high-performance web scraping framework designed for the automated extraction of structured data from websites. It provides a programmable toolkit that manages the complexities of large-scale data collection, including concurrent request orchestration, automatic cookie handling, and robots.txt compliance. By utilizing an asynchronous execution model, the engine maintains high throughput while preventing resource exhaustion during recursive or distributed crawling tasks. The framework is distinguished by its modular, event-driven architecture, which allows developers to hook into sp

This project is a reference library and collection of example code patterns for deploying cloud infrastructure using the AWS CDK. It provides a set of sample projects that demonstrate how to define compute, storage, and networking resources using general purpose programming languages. The library includes reference implementations for various architectural patterns, including serverless backends with GraphQL and WebSocket APIs, container orchestration with load balancers and auto-scaling, and global static website hosting via content delivery networks. It also provides designs for isolated ne

Firecrawl is a web data extraction platform designed to convert unstructured web content into clean, LLM-ready formats like markdown or JSON. It functions as an autonomous web crawler and scraper, capable of mapping entire domains, performing recursive navigation, and executing complex data gathering tasks. By leveraging headless browser orchestration, the system handles dynamic, JavaScript-heavy pages to ensure comprehensive data capture. The platform distinguishes itself through its focus on agentic workflows, providing a programmatic interface that allows autonomous agents to perform live

Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external

ZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure. The platform distinguishes itself through a service-oriented

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It

Erupt is a framework for building administrative interfaces, business intelligence layers, and visual workflow engines. It provides a multi-tenant admin panel and an LLM admin framework that automatically generates web-based management consoles and REST endpoints from backend class definitions. The project distinguishes itself by integrating AI agent orchestration, allowing administrators to manage server operations and execute backend logic through a conversational chat interface. It also features a BI semantic layer that maps raw warehouse data into business-oriented cubes for self-service

Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and contain

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

Gotify is a self-hosted notification server designed to centralize the receipt and dispatch of real-time messages. It provides a RESTful API and a WebSocket gateway, allowing users to programmatically send alerts and push notifications to connected clients. By maintaining a private infrastructure, the platform ensures full control over message history, data retention, and access management. The system distinguishes itself through a modular, plugin-based architecture that allows for the extension of core functionality, including custom HTTP endpoints and webhook event processing. It supports g

This project is a comprehensive sample library and implementation guide for ASP.NET Core. It provides a collection of practical examples and projects that demonstrate how to build web applications, RESTful APIs, and high-performance services. The repository focuses on a variety of architectural patterns, including the development of Minimal APIs, contract-first gRPC services, and real-time communication using WebSockets and Server-Sent Events. It includes detailed implementations for user identity and security, such as token-based authentication and CSRF protection. The codebase covers a bro

DataEase is an open-source, self-hosted business intelligence platform designed for building interactive data visualizations and managing analytical reporting. It provides a centralized environment where users can construct dashboards through a drag-and-drop interface, connecting to diverse data sources including relational databases, data warehouses, and external APIs. The platform distinguishes itself through its focus on embedded analytics and enterprise-grade governance. It allows for the seamless integration of charts, dashboards, and management modules into third-party web applications

Orleans is a .NET distributed actor framework designed for building scalable, cloud-native applications. It implements a virtual actor model where entities with stable identities manage their own state and lifecycle across a cluster of servers. The framework provides a distributed state management system with ACID transaction support and a distributed pub/sub streaming engine for real-time data processing. It distinguishes itself through location-transparent routing, automatic actor activation and deactivation, and elastic cluster scaling that redistributes workloads during node failures. Th

Spinnaker is a multi-cloud continuous delivery platform designed to automate software releases and deployment pipelines across various public cloud providers and Kubernetes clusters. It functions as a cloud deployment orchestrator and infrastructure delivery tool, coordinating the promotion of software artifacts through multiple environments using visual workflows and directed acyclic graphs. The platform distinguishes itself with a dedicated canary analysis engine that compares performance metrics between new and stable software versions to automate release decisions. It utilizes cloud-agnos

Ale is an asynchronous code analysis tool and integrated development environment plugin designed for lightweight text editors. It functions as a language server protocol client, enabling real-time code intelligence and diagnostic feedback by running analysis tasks in the background to ensure the editor interface remains responsive during intensive operations. The plugin utilizes an event-driven architecture to monitor text buffers and trigger linting or formatting routines automatically. It distinguishes itself through a modular extensibility framework that supports a wide range of language-s

NuGetForUnity is a package manager and dependency management system that integrates NuGet packages into Unity projects. It functions as a NuGet client for game development, allowing for the installation, restoration, and updating of external .NET libraries and their transitive dependencies without manual DLL imports. The tool includes a library publishing utility for creating package specification files and uploading custom libraries to remote NuGet servers. It supports the use of both official and custom package feeds, enabling the distribution of proprietary libraries through private regist

This project is a collection of structured study notes and conceptual breakdowns designed for the AWS Certified Cloud Practitioner exam. It serves as a technical reference and study guide, organizing cloud service details and architectural principles to assist in certification preparation. The knowledge base is built using markdown files and includes curated cheat sheets and interactive mind-map visualizations. These tools map complex certification topics into visual hierarchies to enable drill-down study paths and rapid revision. The materials cover a wide range of cloud capabilities, inclu

Quarkus is a Kubernetes-native Java framework designed for building high-performance, memory-efficient applications. It utilizes ahead-of-time native compilation to transform Java code into standalone, optimized binaries that eliminate the need for a virtual machine, enabling rapid startup and reduced memory consumption. By performing code augmentation during the build phase, it shifts heavy processing tasks away from runtime, ensuring that applications are optimized for cloud-native environments. The framework distinguishes itself through a unified approach to reactive and imperative program