28 Repos
Awesome GitHub RepositoriesPre-training Datasets
Large-scale raw text corpora for foundational model training.
Explore 28 awesome GitHub repositories matching part of an awesome list · Pre-training Datasets. Refine with filters or upvote what's useful.
Awesome Pre-training Datasets GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- togethercomputer/redpajama-data
togethercomputer/RedPajama-Data
4,947Auf GitHub ansehenRedPajama-Data ist ein Toolset für das Preprocessing großskaliger Textdatensätze, die zum Training großer Sprachmodelle verwendet werden. Es bietet eine Preprocessing-Pipeline, die sich auf das Bereinigen, Deduplizieren und Bewerten massiver Textsammlungen konzentriert, um Datenqualität und -vielfalt sicherzustellen. Das Projekt nutzt ein Framework zur Bewertung der Dokumentqualität, das Machine Learning und statistische Heuristiken einsetzt, um zu bewerten, ob Dokumente für das Training geeignet sind. Es enthält eine Datensatz-Filter-Pipeline, die Klassifikatoren und Blocklisten verwendet, um unerwünschte Wörter oder URLs zu entfernen. Das System verfügt über ein Text-Deduplizierungstoolset, das redundante Inhalte sowohl mit exakten als auch mit Fuzzy-Matching-Techniken eliminiert. Diese Funktionen ermöglichen die Identifizierung und Entfernung doppelter oder nahezu identischer Dokumente innerhalb eines Korpus.
Reproduced dataset for training large-scale language models.
Python - esbatmop/mnbvc
esbatmop/MNBVC
4,123Auf GitHub ansehenMNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS
Massive, diverse Chinese text corpus from internet sources.
chinesechinese-languagechinese-nlp - luodian/otter
Luodian/Otter
3,410Auf GitHub ansehen🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
Multimodal in-context instruction tuning data.
Python - opengvlab/internvideo
OpenGVLab/InternVideo
2,292Auf GitHub ansehenECCV2024 Video Foundation Models & Data for Multimodal Understanding
Video-centric instruction dataset for chat-based understanding.
Pythonaction-recognitionbenchmarkcontrastive-learning - microsoft/llava-med
microsoft/LLaVA-Med
2,214Auf GitHub ansehenLarge Language-and-Vision Assistant for Biomedicine, built towards multimodal GPT-4 level capabilities.
Biomedical instruction-following dataset for vision-language models.
Python - openmotionlab/motiongpt
OpenMotionLab/MotionGPT
1,938Auf GitHub ansehenNeurIPS 2023 MotionGPT: Human Motion as a Foreign Language, a unified motion-language generation model using LLMs
Human motion-related instruction tuning tasks.
Python3d-generationchatgptgpt - lyuchenyang/macaw-llm
lyuchenyang/Macaw-LLM
1,590Auf GitHub ansehenMacaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration
Multi-turn dialogue dataset for multimodal integration.
Pythondeep-learninglanguage-modelmachine-learning - mbzuai-oryx/video-chatgpt
mbzuai-oryx/Video-ChatGPT
1,504Auf GitHub ansehenACL 2024 🔥 Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
High-quality video instruction dataset for detailed understanding.
Pythonchatbotclipgpt-4 - kakaobrain/coyo-dataset
kakaobrain/coyo-dataset
1,256Auf GitHub ansehenCOYO-700M: Large-scale Image-Text Pair Dataset
Large-scale image-text pairs for pre-training.
Python - cluebenchmark/cluecorpus2020
CLUEbenchmark/CLUECorpus2020
1,012Auf GitHub ansehenLarge-scale Pre-training Corpus for Chinese 100G 中文预训练语料
Cleaned 100GB Chinese corpus for pre-training and NLP tasks.
albertbertchinese - plexpt/chatgpt-corpus
PlexPt/chatgpt-corpus
This project provides a comprehensive Chinese language corpus designed to support the training and fine-tuning of large language models. It serves as a structured natural language processing resource, offering a collection of text data that includes dialogue, customer service interactions, and creative writing. The dataset is organized into distinct thematic categories, allowing for targeted model development across specific conversational and narrative contexts. By providing information in standardized, schema-agnostic text formats, the collection ensures portability across various machine l
Provides a large-scale Chinese language text collection including dialogue and fiction for training large language models.
awesomecorpuscorpus-data - optimalscale/detgpt
OptimalScale/DetGPT
2023-06-13 Added tuned linear weights for Vicuna-7b. 2023-05-25 Our paper is available at this link. 2023-05-09 We have launched our project website. 2023-05-08 The first version of DetGPT is available now! Try our demo.
Query-answer pairs for visual detection and reasoning.
Jupyter Notebook - stevengrove/gpt4tools
StevenGrove/GPT4Tools
GPT4Tools is an intelligent system that can automatically decide, control, and utilize different visual foundation models, allowing the user to interact with images during a conversation.
Tool-use instruction datasets for language models.
Python - phellonchen/x-llm
phellonchen/X-LLM
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
Chinese multimodal instruction dataset for foreign language treatment.
Python - openlamm/lamm
OpenLAMM/LAMM
NeurIPS 2023 Datasets and Benchmarks Track LAMM: Multi-Modal Large Language Models and Applications as AI Agents
Comprehensive dataset for language-assisted multimodal tuning.
Python - fuxiaoliu/lrv-instruction
FuxiaoLiu/LRV-Instruction
ICLR'24 Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Robust instruction tuning to mitigate model hallucinations.
Pythonchatgptevaluationevaluation-metrics - mobvoi/seq-monkey-data
mobvoi/seq-monkey-data
Large-scale dataset used for training general-purpose language models.
- vt-nlp/multiinstruct
VT-NLP/MultiInstruct
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Benchmark dataset for multimodal zero-shot learning.
Python - icoz69/stablellava
icoz69/StableLLAVA
Official repo for StableLLAVA
Synthesized image-dialogue data for instruction tuning.
Python - polyu-chenlab/etbench
PolyU-ChenLab/ETBench
👾 E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024)
Time-sensitive video understanding instruction dataset.
Python
