4 个仓库
Awesome GitHub RepositoriesScraped Data Persistence
Mechanisms for storing unstructured data extracted from web pages into structured databases.
Distinct from Database-Backed Persistence: Existing candidates focus on job metadata or session state, not the persistence of extracted web content.
Explore 4 awesome GitHub repositories matching data & databases · Scraped Data Persistence. Refine with filters or upvote what's useful.
Awesome Scraped Data Persistence GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- lining0806/pythonspidernotes
lining0806/PythonSpiderNotes
7,445在 GitHub 上查看PythonSpiderNotes is a comprehensive instructional resource and framework for building web crawlers and extracting data using the Python programming language. It provides a set of methods for parsing unstructured HTML and JSON data into structured formats for persistent storage. The project includes detailed guides and tutorials on browser automation for retrieving dynamic content, as well as a framework for data extraction. It specifically covers anti-bot bypass techniques, such as rotating proxies and spoofing headers, to avoid IP blocks and detection systems. The capability surface extend
Provides mechanisms for storing unstructured data extracted from web pages into structured databases.
Pythoncaptchacookiepython - kr1s77/python-crawler-tutorial-starts-from-zero
Kr1s77/Python-crawler-tutorial-starts-from-zero
4,599在 GitHub 上查看This project is a Python web scraping tutorial and framework designed for building automated data extraction tools and web crawlers. It provides a structured approach to navigating websites and persisting scraped data to databases. The project includes a toolset for web API analysis, focusing on reverse engineering obfuscated API requests and inspecting network traffic to extract structured data. It also covers optical character recognition workflows to convert visual text within images into machine-readable strings. The framework covers capabilities for headless browser automation to handle
Provides a structured approach to persisting scraped data into databases for long-term storage and analysis.
Python - metatube-community/jellyfin-plugin-metatube
metatube-community/jellyfin-plugin-metatube
4,353在 GitHub 上查看这是一个 Jellyfin 元数据插件和媒体库提供程序，旨在从远程后端服务器获取电影和节目信息。它充当自动化媒体组织者，为 Jellyfin 库填充标题、演员传记和图像。 该插件包含一个本地人脸检测引擎，可自动居中并裁剪演员缩略图和媒体封面。它还提供翻译工具，使用外部 API 将电影摘要和评论转换为目标语言。 该系统通过多碟视频分组、流派重映射和字幕语言标记来管理媒体组织。它支持视觉增强功能，例如在海报上添加字幕指示器以及嵌入远程预告片预览。 后端服务可以通过二进制文件或容器安装，并提供云部署和基于令牌的访问安全选项。
Stores scraped movie and show metadata in SQLite or PostgreSQL to maintain library information across restarts.
C#auto-translationdotnetemby - gnemoug/distribute_crawler
gnemoug/distribute_crawler
3,243在 GitHub 上查看Distribute crawler is a distributed web scraping framework that integrates with Scrapy to coordinate multiple crawler instances across clusters. It utilizes a centralized task queue to manage and scale concurrent data collection operations, enabling horizontal scaling of scraping tasks across multiple worker nodes. The framework distinguishes itself through its focus on large-scale data management and traffic control. It persists scraped items and binary assets into document-oriented database clusters, utilizing deduplication logic to optimize bandwidth and storage. To maintain consistent dat
Maps extracted web content directly into flexible document-oriented database schemas for scalable storage.
Python
