Awesome GitHub RepositoriesData Exporters

MediaCrawler is an automated web scraping framework designed to extract public posts, comments, and creator metadata from various social media platforms. It functions as a headless browser automator, utilizing real browser instances to render dynamic content and execute the client-side scripts necessary for interacting with modern web interfaces. The system distinguishes itself through a focus on session persistence and network flexibility. It supports remote debugging to reuse active browser sessions and cookies, which helps minimize the risk of triggering platform security challenges. To ma

labelImg est un outil d'annotation d'images de bureau et un utilitaire de préparation de jeux de données utilisé pour créer des jeux de données étiquetés pour l'entraînement en vision par ordinateur. Il fournit une interface graphique pour dessiner des boîtes englobantes autour des objets dans les images et leur assigner des étiquettes de classe pour construire des données de vérité terrain pour les modèles d'apprentissage automatique. Le logiciel prend spécifiquement en charge le format d'annotation XML Pascal VOC, exportant les coordonnées des images et les noms de classe dans des structures XML ou texte standard. Il permet aux utilisateurs de charger des listes de classes prédéfinies à partir de fichiers texte pour standardiser la dénomination sur l'ensemble d'un projet. Au-delà de l'étiquetage initial, l'outil couvre les flux de travail d'annotation d'images, y compris la visualisation des annotations enregistrées et la vérification manuelle des jeux de données. Cela inclut la possibilité de marquer les images comme vérifiées ou difficiles pour maintenir la qualité du jeu de données.

Pulumi is an infrastructure-as-code framework that enables the definition, deployment, and management of cloud resources using general-purpose programming languages. It functions as a cloud resource orchestrator that coordinates the lifecycle of heterogeneous infrastructure by executing code to construct dependency graphs and reconciling the desired state against actual cloud environments. The platform distinguishes itself through a language-host runtime bridge that allows developers to use standard programming languages to define infrastructure, rather than relying solely on domain-specific

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

Automa is a browser-based automation platform that enables users to build, schedule, and execute repetitive web tasks through a visual, no-code interface. By operating as a browser extension, it provides a canvas-based environment where users construct workflows by connecting functional blocks to interact with web elements, manage browser state, and process data. The platform distinguishes itself through its deep integration with the browser environment, allowing for complex orchestration such as event-driven triggers, cross-origin request handling, and the ability to package workflows as sta

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

Twint is an open-source intelligence and data extraction framework designed to gather public social media information. It functions as a command-line utility that retrieves posts, user profiles, and follower lists directly from web interfaces, bypassing the need for official platform developer credentials or authentication keys. The tool distinguishes itself by enabling automated, large-scale data collection through terminal-based orchestration. It supports granular filtering by keywords, geographic locations, time ranges, and account status, allowing researchers to build targeted datasets fo

Hammerspoon is a programmable automation engine for macOS that enables deep system-level control through a Lua scripting environment. By bridging high-level scripts with native Objective-C APIs, it allows users to interact with the operating system's accessibility tree, intercept hardware input streams, and manage the lifecycle of running applications. The project distinguishes itself through an event-driven architecture that registers asynchronous hooks for system notifications and hardware events. This allows for real-time automation, such as remapping keyboard and mouse inputs, managing wi

Dask est un framework de calcul parallèle et un planificateur de tâches distribué conçu pour mettre à l'échelle les flux de travail de science des données Python, des machines uniques aux grands clusters. Il fonctionne comme un gestionnaire de ressources de cluster qui orchestre la logique computationnelle en représentant les tâches et leurs dépendances sous forme de graphes acycliques dirigés. Cette architecture permet au système d'automatiser la distribution des charges de travail sur le matériel disponible tout en gérant des exigences d'exécution complexes. Le projet se distingue par un moteur d'évaluation paresseuse qui diffère les opérations sur les données jusqu'à ce qu'elles soient explicitement demandées, permettant une optimisation globale du graphe et une allocation efficace des ressources. Il intègre le déversement de données conscient de la mémoire pour éviter les plantages du système lors du traitement de jeux de données dépassant la mémoire disponible, et il utilise la fusion de graphes de tâches pour combiner des séquences d'opérations en étapes d'exécution uniques, minimisant la surcharge de planification et la communication entre nœuds. La plateforme fournit une surface de capacités complète pour l'analyse de données à grande échelle, incluant le support pour l'apprentissage automatique distribué, l'intégration du calcul haute performance et le traitement de données parallèle. Elle offre des outils étendus pour la gestion du cycle de vie des clusters, le profilage des performances et la surveillance en temps réel de l'exécution des tâches. Les utilisateurs peuvent déployer ces environnements sur diverses infrastructures, incluant le matériel local, les fournisseurs cloud, les systèmes conteneurisés et les clusters de calcul haute performance.

KeeWeb is a web-based password manager and vault that allows users to open and edit encrypted databases through a browser interface. It functions as a cross-platform tool for managing password vaults using the KeePass database format. The application provides a self-hosted password vault that can be deployed as a single HTML file or via Docker. It integrates with remote storage providers using OAuth to synchronize encrypted database files across multiple devices. The system includes capabilities for secure credential generation, two-factor authentication management through time-based one-tim

This is a tool for searching, downloading, and archiving articles and engagement metadata from WeChat official accounts. It functions as a web-based content scraper and data exporter, allowing for the automated retrieval of social media content and the collection of performance metrics. The project distinguishes itself through a system that captures session credentials and authentication cookies from desktop clients via a local proxy to access private engagement data. It utilizes a concurrent proxy-pool fetching mechanism to download large volumes of content while avoiding rate limits, and it

This project is an open-source intelligence reconnaissance framework and recursive attack surface mapper. It functions as a containerized security scanner designed to map public-facing infrastructure, perform subdomain enumeration, and automate the gathering of open-source intelligence. The system employs a recursive discovery engine to iteratively explore target infrastructure, utilizing a plugin-based module architecture to extend scanning capabilities. It integrates third-party APIs for data enrichment and applies YARA rules across discovered assets to identify specific vulnerability patte

This project is an educational resource and a collection of instructional materials for performing data manipulation and statistical analysis using Python. It provides a comprehensive set of guides and code examples for using the Pandas, NumPy, and Matplotlib libraries to analyze structured data. The resource includes a dedicated guide for reshaping, cleaning, and aggregating tabular data and time series via Pandas, alongside a reference for high-performance vectorized operations and linear algebra using NumPy. It also features tutorials for creating publication-quality charts, distribution p

unopim is an AI-powered product information management system that serves as a centralized repository for managing product attributes, categories, and variations. It functions as a containerized product repository and a multi-channel data distributor, synchronizing consistent product information and pricing across diverse external sales platforms and marketplaces. The platform distinguishes itself through an LLM-based catalog manager that provides a conversational interface for executing data management tasks. This allows users to perform item creation, content enrichment, and quality scans u

Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin

Gyroflow is a gyroscope video stabilization software and IMU telemetry processor designed to remove camera shake from video files. It functions as a hardware-accelerated video renderer and lens calibration tool, utilizing embedded or external gyroscope and accelerometer data to perform pixel-level stabilization. The system is distinguished by its ability to integrate with professional non-linear video editing software via plugins, allowing stabilization to be applied directly to timelines without transcoding original footage. It supports diverse telemetry ingestion from camera brands, flight

WechatExporter is a chat backup conversion tool designed to extract WeChat chat records from backups and convert them into human-readable text, HTML, or PDF files for archiving. It functions as a data extraction system that transforms proprietary messaging database backups into standardized document formats. The project includes a template-based document generator that allows users to modify the visual layout and formatting of exported conversation records. It also features an incremental export system that tracks and saves only new messages sent since the previous export to prevent data dupl

This repository is a comprehensive collection of functional 2D and 3D demo projects and implementation samples for the Godot Game Engine. It serves as an interactive tutorial and reference library, providing a working codebase to demonstrate how to apply engine features in real-world scenarios. The collection focuses on practical implementation guides, covering a wide array of technical capabilities from basic engine fundamentals to advanced rendering and scripting techniques. It allows users to study the application of node-based composition, asset pipelines, and game logic through direct ex

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

elasticsearch-dump is a command line tool for importing, exporting, and transferring data between Elasticsearch and OpenSearch instances. It functions as an index dump utility that saves documents, mappings, and analyzers to local files or standard output. The tool enables the movement of data between clusters using local files as an intermediary and can flatten nested JSON documents into CSV files for external analysis. It allows for the modification or anonymization of documents during the transfer process through the use of custom JavaScript functions. The utility covers data extraction a