14 个仓库
Sequential processing of tasks using a queue to manage system resource consumption.
Distinct from Batch Processing: Focuses on the queueing mechanism for file processing rather than bulk data operations
Explore 14 awesome GitHub repositories matching data & databases · Task Queues. Refine with filters or upvote what's useful.
ImageOptim is a macOS image optimizer and GUI image compressor designed to reduce image file sizes for web and disk storage. It functions as a lossless image optimizer that removes unnecessary metadata and optimizes encoding to reduce file sizes without losing pixel data. The application handles macOS media asset management and web image preparation by shrinking files to improve website loading speeds and reduce storage usage. It provides both lossless and lossy compression options to maintain visual quality while reducing the disk space used by images.
Manages a queue of images to process them sequentially and prevent system resource exhaustion.
TubeArchivist is a self-hosted YouTube video archiving system and metadata indexer. It functions as a personal media library and download manager that allows users to create a searchable offline collection of videos, channels, and playlists. The system distinguishes itself by indexing subtitles, comments, and channel information for full-text search and retrieval. It features automated media synchronization to track subscriptions and playlists, ensuring new content is automatically queued and downloaded as it is published. The project provides a broad set of capabilities for digital asset ma
Processes pending items in a sequential download queue to retrieve files for offline storage.
downkyicore is a cross-platform desktop application designed for the automated batch downloading of Bilibili videos, including support for high-resolution 8K and HDR content. It functions as a batch media downloader that manages multiple files and collections simultaneously through an automated queue system. The project integrates FFmpeg for media processing to separate audio and video streams and remove watermarks. It uses specialized mechanisms to bypass access restrictions and rate limits by employing random identifiers and mobile referrers during the parsing process. The system includes
Manages a sequential queue of download tasks using a local database to handle batch processing and retries.
Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp
Enqueues scan tasks for later execution with configurable concurrency limits.
This repository is a collection of YAML automations and scripts for Home Assistant that coordinate devices and services into context‑aware home routines. Its design centers on combining multiple sensor inputs—motion, bed presence, garage door state—into a single occupancy signal that gates automations, reducing false triggers. The collection delivers actionable mobile notifications with script‑bound buttons for snoozing or escalating alerts, and integrates an external vision service to classify camera snapshots so package deliveries are detected and announced. What distinguishes this configur
Processes tasks one at a time from a persistent queue with per-task time thresholds and delay guards.
Parabolic is a graphical frontend for the yt-dlp engine, serving as a web media downloader and extractor. It provides a visual interface for saving high-quality video and audio content from various web platforms into local files. The application functions as a multi-format media exporter, allowing content to be saved into diverse audio and video containers. It includes a media metadata manager to capture and store associated information, such as subtitles and technical metadata, alongside the downloaded files. The system supports batch content acquisition through concurrent download manageme
Coordinates multiple simultaneous download tasks using a worker queue to optimize system resource consumption.
这是一个用于自动语音识别的 Windows 应用程序,将视频文件中的口语转录为带时间戳的 SRT 字幕文件。它作为一个字幕生成器和翻译工具,将媒体语音转换为同步文本。 该软件作为一个批量媒体转录器,允许同时处理多个音频和视频文件以批量生成字幕。它包括一个翻译工作流,用于在不同语言间转换转录内容,以创建双语或本地化文件。 该系统还提供文本优化能力,使用正则表达式和自定义过滤器通过移除填充词和不需要的模式来清理转录内容。此功能由原生 Windows 图形用户界面支持。
Uses a task queue to process multiple media files sequentially and manage system resources.
Rudder Server is a customer data platform and event routing pipeline designed to collect, transform, and route customer event data from various sources to data warehouses and business tools. It functions as a customer identity resolver, linking identifiers from multiple sources to build a unified identity graph and comprehensive behavioral customer profiles. The system differentiates itself through reverse ETL capabilities, which push processed customer segments and audiences from data warehouses back into operational third-party applications. It also provides a containerized data plane for K
Schedules bulk data uploads and synchronization tasks using a priority queue to manage system load.
UniExtract2 是一套专为通用归档提取、批量解压和文件格式分析而设计的工具。它将文件从各种压缩格式、软件安装程序、磁盘镜像和游戏归档中提取到本地目录。 该项目包含一个文件格式分析器,无需完全提取即可通过扫描内部内容和 Header 来识别文件类型。它还具有一个归档密码解密器,尝试使用预定义的常用密码列表恢复对受保护归档的访问。 该工具通过批处理队列处理和递归目录提取支持批量解压工作流,以处理嵌套归档结构。它进一步提供了在不执行安装过程的情况下分析软件安装程序和磁盘镜像内部内容的能力。
Uses a task queue to manage the sequential decompression of multiple archive files.
Abogen is a text-to-speech audiobook generator that transforms digital documents and subtitle files into audiobooks. It utilizes language models to perform text normalization, rewriting contractions and punctuation to produce more natural speech synthesis. The system features a voice profile mixer that blends multiple voice models using adjustable weight ratios to create personalized synthetic voices. It also includes an automated export system that sends completed audio files and metadata to a remote Audiobookshelf server via a web API. The project manages the end-to-end audiobook productio
Manages a queue of document processing tasks with support for per-file configuration overrides.
Ripme is a batch media downloader and web media scraper designed for extracting images and videos from image-hosting platforms and social media sites. It functions as an image gallery downloader and a network client capable of retrieving full albums and paginated content. The project includes a custom media ripper framework that allows for the definition of new extraction rules to support websites lacking native support. It features a proxy-enabled network layer for routing requests through HTTP or SOCKS servers and supports session-based content retrieval using authentication cookies and cus
Scans web pages for linked galleries and adds them to a download queue for batch processing.
Neo-Backup is an Android application backup tool and rooted device data manager. It creates encrypted copies of application data and system files on rooted devices to preserve application state and settings. The project functions as a backup scheduler, allowing for the automation of periodic data preservation tasks based on custom application lists and fixed timetables. To protect sensitive information, it utilizes password-based encryption to secure backup archives. The system provides capabilities for device-wide data backup and recovery, including the ability to restore applications and t
Implements a sequential task queue to manage multiple data extraction jobs without overloading device resources.
ncmdump is a proprietary audio cache converter and binary stream decryption utility. It decrypts raw audio data from specialized music cache files and transforms them into standard audio formats. The project functions both as a standalone tool and a cross-language conversion library. It exposes its internal decryption and conversion logic as a dynamic library, allowing the capabilities to be embedded into external applications written in different programming languages. The utility provides batch media processing through recursive directory traversal, enabling the identification and conversi
Manages a queue of target files for sequential processing to optimize resource usage.
TubeSync is a containerized media management tool and self-hosted archiver designed to automate the downloading and organization of video content from online sources, such as YouTube channels and playlists, into a local library for offline access. It functions as a download manager and metadata generator, utilizing a web interface to manage video subscriptions and synchronization settings. The system features a rule-based content filter that evaluates video metadata against user-defined conditions to determine which items enter the download queue. To handle restricted or age-gated content, it
Manages download tasks in a database, processed sequentially by a worker process with real-time updates.