Projects sharing features with Img2dataset

Data engineering practice repository providing tutorials, distributed processing engines, and Python data pipeline automation scripts. The system encompasses automated data validation, distributed compute aggregation, embedded columnar querying, lazy evaluation planning, partitioned storage export, and cloud storage retrieval. The capability surface covers cloud integration and storage, data engineering and pipelines, data processing and analytics, data quality and testing, database and storage, file management, and monitoring and observability.

Pandarallel is a Python library designed to accelerate data manipulation workflows by distributing computationally intensive tasks across all available CPU cores. It functions as a parallel processing utility that integrates directly into standard data analysis pipelines to reduce the execution time of large-scale operations. The library employs a wrapper pattern that intercepts standard data processing calls, redirecting them from sequential execution to a parallelized architecture. It manages this distribution through a multiprocessing-based task model, which utilizes shared-memory serializ

mmcv is a foundation library for computer vision based on PyTorch. It provides a comprehensive system for constructing convolutional neural networks, a toolkit for image and video preprocessing, and a collection of high-performance deep learning vision operators. The project is distinguished by its hardware-accelerated kernels for complex operations such as deformable convolutions and region pooling. It features a configuration-driven framework that allows for the dynamic instantiation of network layers and the registration of custom modules without modifying code. The library covers a broad

This project is a command-line synchronization client for OneDrive and SharePoint libraries on Linux. It functions as a synchronization engine that aligns local filesystems with cloud storage through bidirectional, unidirectional, or download-only workflows. The client supports headless authentication for servers without web browsers and can be deployed as a background service or within a containerized environment. It enables the management of multiple distinct cloud accounts on a single system and integrates with shared SharePoint sites and document libraries. The synchronization engine inc

face.evoLVe is a deep learning library designed for the training and deployment of facial recognition models. It provides a comprehensive framework for converting facial images into numerical feature vectors, enabling identity verification and similarity analysis across large-scale datasets. The project facilitates the entire lifecycle of facial analysis, from dataset preparation and image standardization to distributed model training. It includes utilities for detecting facial landmarks and applying geometric transformations to ensure consistent input orientation, as well as data augmentatio

aws-sdk-pandas is a Python library that integrates pandas dataframes with AWS services, acting as a cloud data ETL tool and data lake connector. It provides a unified interface to move and transform data between in-memory dataframes and cloud storage, databases, and data warehouses. The project distinguishes itself as a distributed compute orchestrator capable of submitting pandas-based workloads to EMR clusters and serverless processing environments. It further specializes in coordinating distributed data processing via Ray cluster initialization to handle datasets that exceed the memory of

Okdownload is a multi-threaded HTTP download engine and transfer library designed for concurrent file downloading. It functions as a resumable file downloader that splits single files into multiple blocks for simultaneous processing to increase network throughput and transfer speeds. The system distinguishes itself through a builder-based component injection framework and a strategy pattern that allows for custom download engines. Developers can replace default storage and connection modules or override the logic used for block splitting and filename generation. The library provides comprehe

This project is a deep learning system designed for real-time emotion recognition and facial expression analysis. It utilizes a convolutional neural network architecture to process raw visual input, mapping complex facial patterns to seven distinct emotional states through a supervised machine learning pipeline. The system functions as both a training framework and an inference engine. It includes utilities for preparing and standardizing large image datasets to ensure consistent input quality, alongside a real-time processing pipeline that captures and buffers live video frames to perform co

This project is a distributed headless Chrome web crawler and data extraction framework. It functions as a JavaScript rendering engine that uses a headless browser to process dynamic pages, extracting structured data from websites that require JavaScript execution. The system is designed for scalable data collection across multiple nodes, using distributed task synchronization and shared caches to prevent duplicate work. It distinguishes itself through the ability to emulate specific client environments by configuring user agents and viewport dimensions, while capturing visual evidence such a

This project is a machine learning data pipeline designed to automate the collection, curation, and preparation of large-scale image datasets. It functions as an image dataset scraper and computer vision curator, providing the necessary infrastructure to aggregate categorized files from web sources and organize them into structured directories for model development. The system distinguishes itself through a batch-processing architecture that integrates data acquisition with automated integrity validation. By scanning files to remove corrupted or invalid images and applying deterministic parti

SynapseML is an Apache Spark machine learning library designed for building and scaling machine learning workflows and data pipelines across distributed clusters. It serves as a distributed machine learning pipeline framework and a distributed inference engine for executing hardware-accelerated predictions and deep learning tasks on large-scale datasets. The project functions as a cloud AI integration layer, allowing users to apply pretrained artificial intelligence services for text, vision, and speech within distributed pipelines. It also includes a dedicated suite of tools for distributed

This project is a collection of Python implementations for web scraping, network traffic interception, data analysis, and sentiment analysis. It provides methods for extracting structured data from websites and mobile application interfaces. The collection includes tools for capturing and analyzing network packets from mobile applications to identify hidden internal API endpoints. It also features scripts for evaluating the emotional tone and public perception of text data. The project covers data manipulation and transformation of large datasets, as well as the generation of charts and grap

ECommerceCrawlers is an educational collection of Python-based crawler scripts designed to extract data from a variety of public websites, including e-commerce platforms, social media sites, news outlets, and multimedia sources. The project serves as a learning resource for web scraping techniques, offering ready-to-run examples that demonstrate practical data extraction methods. The toolkit covers a broad range of data types, including product listings and prices from online retail platforms, public posts and profiles from social networking sites, articles from news and blogging platforms, p

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in

Joblib is a suite of utilities for parallelizing computational workloads and optimizing the storage of large numerical datasets and function results. It functions as a parallel computing library and multiprocessing wrapper that distributes function execution across multiple CPU cores to accelerate independent tasks and computational loops. The project provides a disk caching framework that persists expensive function outputs to the filesystem, re-evaluating them only when input arguments change. It further specializes in the serialization of large numerical arrays, utilizing efficient compres

Augmentor is a Python image augmentation library and framework designed to expand machine learning datasets. It functions as a preprocessing tool that generates synthetic image variations to increase data diversity and as a training data streamer that feeds augmented images and labels directly into neural network loops without requiring intermediate disk storage. The framework maintains spatial alignment between images and their corresponding masks, which is required for semantic segmentation training. It supports various geometric and pixel-level transformations, including elastic distortion

This project is a Model Context Protocol server that connects large language models to web scraping and crawling tools. It functions as a bridge, allowing LLM clients to utilize a web crawling engine and scraping utilities to extract and process web data. The server integrates a markdown web converter that transforms dynamic web pages and PDF documents into clean markdown to optimize consumption by AI models. It also provides a browser automation interface for controlling headless sessions and bypassing access restrictions. The system covers broad capabilities including large-scale website d

NSFW detection on the client-side via TensorFlow.js

This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal language models. It provides a comprehensive suite for vision-language pretraining, enabling the development of models that map images and text into shared latent spaces. The framework is distinguished by its capabilities in high-fidelity image generation and multimodal research, utilizing normalizing flows and variational autoencoders to produce images from text prompts or class labels. It supports the development of both generative and contrastive models, allowing for a wide

Cyberduck is a multi-protocol file transfer client and cross-platform file manager for Mac and Windows. It functions as a cloud storage manager and remote storage mounter, allowing users to upload, organize, and synchronize data between local drives and remote endpoints. The application provides a unified interface for managing files across FTP, SFTP, WebDAV, S3, and other cloud storage protocols, including Amazon S3, Backblaze B2, Microsoft Azure, and OneDrive. It distinguishes itself with a client-side encryption vault that encrypts files and folders locally before they are transmitted to r

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites. The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem. The system utilizes a pipeline-based data

This project is a neural network image classifier and a set of tools for building and training convolutional neural networks to recognize and categorize images. It serves as a machine learning educational guide, providing a practical resource for learning neural network fundamentals through an onboarding process. The system includes a dedicated workflow for pretrained model fine-tuning, allowing existing network weights to be adapted to new image categories. This is supported by a transfer learning pipeline that replaces final classification layers and adjusts weights through targeted retrain

This project is an AWS pandas integration library and data pipeline framework designed to simplify the movement and transformation of data between local memory and AWS storage and analytics services. It functions as a cloud data lake toolkit and storage file manager, allowing users to read, write, and transform structured data across various cloud environments. The library distinguishes itself as a distributed compute orchestrator capable of managing clusters in environments such as EMR to process datasets that exceed the memory limits of a single machine. It also provides specialized capabil

FlutterFire is a collection of official plugins that integrate Firebase backend services into Flutter applications. It serves as a backend-as-a-service integration library, providing client-side wrappers for cloud authentication, databases, storage, and monitoring services. The project enables the integration of serverless backend logic and real-time data synchronization using NoSQL documents and state synchronization. It also provides capabilities for generative AI integration, including large language models, image generation, and local machine learning model management. The suite covers a

JMComic-Crawler-Python is a high-performance asynchronous web scraper and API client designed to programmatically retrieve images and metadata from a comic hosting service. It functions as a media archiving tool for batch downloading albums and chapters, automating the process of saving content to a local filesystem. The project is distinguished by its ability to reverse server-side pixel obfuscation, using a decryption tool to reconstruct sliced and shuffled images. To maintain stable connectivity, it utilizes a network bypass utility featuring dynamic domain rotation and proxy routing to ci

This project is a JavaScript library and SDK used to integrate web and mobile applications with cloud services. It serves as a bridge to backend providers for user authentication, binary object storage, and real-time data synchronization. The library provides a unified interface for managing cloud identity and access, executing queries and mutations against GraphQL endpoints, and consuming REST APIs with secure request signing. It also includes tools for accessing machine learning services for natural language processing and computer vision. Broad capability areas include offline-first data