14 dépôts
Sequential processing of tasks using a queue to manage system resource consumption.
Distinct from Batch Processing: Focuses on the queueing mechanism for file processing rather than bulk data operations
Explore 14 awesome GitHub repositories matching data & databases · Task Queues. Refine with filters or upvote what's useful.
ImageOptim is a macOS image optimizer and GUI image compressor designed to reduce image file sizes for web and disk storage. It functions as a lossless image optimizer that removes unnecessary metadata and optimizes encoding to reduce file sizes without losing pixel data. The application handles macOS media asset management and web image preparation by shrinking files to improve website loading speeds and reduce storage usage. It provides both lossless and lossy compression options to maintain visual quality while reducing the disk space used by images.
Manages a queue of images to process them sequentially and prevent system resource exhaustion.
TubeArchivist is a self-hosted YouTube video archiving system and metadata indexer. It functions as a personal media library and download manager that allows users to create a searchable offline collection of videos, channels, and playlists. The system distinguishes itself by indexing subtitles, comments, and channel information for full-text search and retrieval. It features automated media synchronization to track subscriptions and playlists, ensuring new content is automatically queued and downloaded as it is published. The project provides a broad set of capabilities for digital asset ma
Processes pending items in a sequential download queue to retrieve files for offline storage.
downkyicore is a cross-platform desktop application designed for the automated batch downloading of Bilibili videos, including support for high-resolution 8K and HDR content. It functions as a batch media downloader that manages multiple files and collections simultaneously through an automated queue system. The project integrates FFmpeg for media processing to separate audio and video streams and remove watermarks. It uses specialized mechanisms to bypass access restrictions and rate limits by employing random identifiers and mobile referrers during the parsing process. The system includes
Manages a sequential queue of download tasks using a local database to handle batch processing and retries.
Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp
Enqueues scan tasks for later execution with configurable concurrency limits.
This repository is a collection of YAML automations and scripts for Home Assistant that coordinate devices and services into context‑aware home routines. Its design centers on combining multiple sensor inputs—motion, bed presence, garage door state—into a single occupancy signal that gates automations, reducing false triggers. The collection delivers actionable mobile notifications with script‑bound buttons for snoozing or escalating alerts, and integrates an external vision service to classify camera snapshots so package deliveries are detected and announced. What distinguishes this configur
Processes tasks one at a time from a persistent queue with per-task time thresholds and delay guards.
Parabolic is a graphical frontend for the yt-dlp engine, serving as a web media downloader and extractor. It provides a visual interface for saving high-quality video and audio content from various web platforms into local files. The application functions as a multi-format media exporter, allowing content to be saved into diverse audio and video containers. It includes a media metadata manager to capture and store associated information, such as subtitles and technical metadata, alongside the downloaded files. The system supports batch content acquisition through concurrent download manageme
Coordinates multiple simultaneous download tasks using a worker queue to optimize system resource consumption.
Ceci est une application Windows pour la reconnaissance vocale automatique qui transcrit l'audio parlé à partir de fichiers vidéo en fichiers de sous-titres SRT horodatés. Elle sert de générateur de sous-titres et d'outil de traduction qui convertit la parole des médias en texte synchronisé. Le logiciel fonctionne comme un transcripteur de médias par lots, permettant le traitement simultané de plusieurs fichiers audio et vidéo pour générer des sous-titres en masse. Il inclut un workflow de traduction pour convertir les transcriptions entre différentes langues pour la création de fichiers bilingues ou localisés. Le système fournit également des capacités d'affinage de texte, utilisant des expressions régulières et des filtres personnalisés pour nettoyer les transcriptions en supprimant les mots de remplissage et les motifs indésirables. Ceci est supporté par une interface utilisateur graphique Windows native.
Uses a task queue to process multiple media files sequentially and manage system resources.
Rudder Server est une plateforme de données client et un pipeline de routage d'événements conçu pour collecter, transformer et router les données d'événements client depuis diverses sources vers des entrepôts de données et des outils métier. Il fonctionne comme un résolveur d'identité client, liant les identifiants de multiples sources pour construire un graphe d'identité unifié et des profils client comportementaux complets. Le système se différencie par des capacités de reverse ETL, qui poussent les segments et audiences client traités depuis les entrepôts de données vers des applications tierces opérationnelles. Il fournit également un plan de données conteneurisé pour les déploiements Kubernetes, permettant la gestion de l'infrastructure de données en tant que code. La plateforme couvre un large éventail de capacités de gestion de données, y compris la transformation d'événements en temps réel, la validation de schéma via des catalogues de données et la gouvernance de la confidentialité. Celles-ci incluent des outils pour gérer le consentement des utilisateurs, appliquer la résidence des données au sein de régions géographiques spécifiques et masquer les informations personnellement identifiables en transit. L'installation et le déploiement des composants du plan de données sont gérés à l'aide de charts Helm.
Schedules bulk data uploads and synchronization tasks using a priority queue to manage system load.
UniExtract2 est une suite d'outils conçue pour l'extraction universelle d'archives, la décompression par lots et l'analyse de formats de fichiers. Elle récupère des fichiers à partir de divers formats compressés, installateurs de logiciels, images disque et archives de jeux dans des répertoires locaux. Le projet inclut un analyseur de format de fichier qui identifie les types de fichiers en scannant les contenus internes et les en-têtes sans nécessiter une extraction complète. Il dispose également d'un décrypteur de mot de passe d'archive qui tente de récupérer l'accès aux archives protégées en utilisant une liste prédéfinie de mots de passe courants. L'outil prend en charge les workflows de décompression en masse via le traitement par file d'attente par lots et l'extraction récursive de répertoires pour gérer les structures d'archives imbriquées. Il fournit en outre la capacité d'analyser les contenus internes des installateurs de logiciels et des images disque sans exécuter le processus d'installation.
Uses a task queue to manage the sequential decompression of multiple archive files.
Abogen is a text-to-speech audiobook generator that transforms digital documents and subtitle files into audiobooks. It utilizes language models to perform text normalization, rewriting contractions and punctuation to produce more natural speech synthesis. The system features a voice profile mixer that blends multiple voice models using adjustable weight ratios to create personalized synthetic voices. It also includes an automated export system that sends completed audio files and metadata to a remote Audiobookshelf server via a web API. The project manages the end-to-end audiobook productio
Manages a queue of document processing tasks with support for per-file configuration overrides.
Ripme is a batch media downloader and web media scraper designed for extracting images and videos from image-hosting platforms and social media sites. It functions as an image gallery downloader and a network client capable of retrieving full albums and paginated content. The project includes a custom media ripper framework that allows for the definition of new extraction rules to support websites lacking native support. It features a proxy-enabled network layer for routing requests through HTTP or SOCKS servers and supports session-based content retrieval using authentication cookies and cus
Scans web pages for linked galleries and adds them to a download queue for batch processing.
Neo-Backup is an Android application backup tool and rooted device data manager. It creates encrypted copies of application data and system files on rooted devices to preserve application state and settings. The project functions as a backup scheduler, allowing for the automation of periodic data preservation tasks based on custom application lists and fixed timetables. To protect sensitive information, it utilizes password-based encryption to secure backup archives. The system provides capabilities for device-wide data backup and recovery, including the ability to restore applications and t
Implements a sequential task queue to manage multiple data extraction jobs without overloading device resources.
ncmdump is a proprietary audio cache converter and binary stream decryption utility. It decrypts raw audio data from specialized music cache files and transforms them into standard audio formats. The project functions both as a standalone tool and a cross-language conversion library. It exposes its internal decryption and conversion logic as a dynamic library, allowing the capabilities to be embedded into external applications written in different programming languages. The utility provides batch media processing through recursive directory traversal, enabling the identification and conversi
Manages a queue of target files for sequential processing to optimize resource usage.
TubeSync is a containerized media management tool and self-hosted archiver designed to automate the downloading and organization of video content from online sources, such as YouTube channels and playlists, into a local library for offline access. It functions as a download manager and metadata generator, utilizing a web interface to manage video subscriptions and synchronization settings. The system features a rule-based content filter that evaluates video metadata against user-defined conditions to determine which items enter the download queue. To handle restricted or age-gated content, it
Manages download tasks in a database, processed sequentially by a worker process with real-time updates.