Awesome GitHub RepositoriesHTTP Control Interfaces

gpt-crawler is a web scraping utility designed to extract website content and convert it into structured text files for use as AI model knowledge bases. It functions as a data generator that crawls specified web addresses to produce the knowledge files required for building custom GPTs, grounding large language models, and providing context to AI agents. The system transforms raw HTML into clean Markdown text to reduce token usage and improve readability for AI models. It utilizes token-aware content chunking and output file size limitations to ensure generated datasets remain compatible with

This project is a Sina Weibo web scraper and social media data pipeline designed to extract user profiles, posts, comments, and multimedia assets. It functions as a containerized data crawler that automates the collection and local storage of social media content and engagement metrics. The system includes a processing layer that uses large language models to analyze scraped text, generating summaries and sentiment analysis. It differentiates itself through a deployment-ready container model featuring an HTTP interface for managing extraction tasks and monitoring job progress. The crawler co

Automatic Optical Disc Ripping Server is a headless system that detects inserted CDs, DVDs, and Blu-rays to automatically extract media, transcode video, and eject discs. It functions as a multi-drive media digitizer using a concurrent processing pipeline to rip and transcode media from several optical drives simultaneously without queuing. The system includes an asynchronous video transcoding pipeline that batches conversion tasks to run during scheduled off-peak hours. It also serves as a media server automation tool, fetching metadata from online APIs to name folders and trigger library re