How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
MNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS
This project provides a comprehensive Chinese language corpus designed to support the training and fine-tuning of large language models. It serves as a structured natural language processing resource, offering a collection of text data that includes dialogue, customer service interactions, and creative writing. The dataset is organized into distinct thematic categories, allowing for targeted model development across specific conversational and narrative contexts. By providing information in standardized, schema-agnostic text formats, the collection ensures portability across various machine l
This project is a development course and learning curriculum focused on building large language model chatbots. It provides a structured series of tutorials for creating conversational agents through the application of natural language processing and deep learning models. The materials include a technical walkthrough for implementing neural networks and word embeddings to handle automated question-answering tasks. It also provides a guide for constructing large-scale conversation corpora from external text sources to train and evaluate dialogue systems. The curriculum covers core text analys
Question answering dataset featured in "Teaching Machines to Read and Comprehend
Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料
The main features of cluebenchmark/cluecorpus2020 are: Datasets and Corpora, Pre-training Datasets.
Projects with overlapping indexed features include: esbatmop/mnbvc — MNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and… plexpt/chatgpt-corpus — This project provides a comprehensive Chinese language corpus designed to support the training and fine-tuning of… chinawithfrank/chatbotcourse — This project is a development course and learning curriculum focused on building large language model chatbots. It… deepmind/rc-data — Question answering dataset featured in "Teaching Machines to Read and Comprehend. goldsmith/wikipedia — Wikipedia. fuxiaoliu/lrv-instruction — [ICLR'24] Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning.