28 repository-uri
Awesome GitHub RepositoriesPre-training Datasets
Large-scale raw text corpora for foundational model training.
Explore 28 awesome GitHub repositories matching part of an awesome list · Pre-training Datasets. Refine with filters or upvote what's useful.
Awesome Pre-training Datasets GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- togethercomputer/redpajama-data
togethercomputer/RedPajama-Data
4,947Vezi pe GitHubRedPajama-Data este un set de instrumente pentru preprocesarea seturilor de date text la scară largă utilizate pentru antrenarea modelelor de limbaj mari (LLM). Oferă un pipeline de preprocesare axat pe curățarea, deduplicarea și punctarea unor colecții masive de text pentru a asigura calitatea și diversitatea datelor. Proiectul utilizează un framework de punctare a calității documentelor care folosește machine learning și euristici statistice pentru a evalua dacă documentele sunt potrivite pentru antrenare. Include un pipeline de filtrare a seturilor de date care utilizează clasificatori și liste de blocare pentru a elimina cuvintele sau URL-urile nedorite. Sistemul dispune de un set de instrumente de deduplicare a textului care elimină conținutul redundant folosind tehnici de potrivire exactă și fuzzy. Aceste capabilități permit identificarea și eliminarea documentelor duplicate sau aproape identice dintr-un corpus.
Reproduced dataset for training large-scale language models.
Python - esbatmop/mnbvc
esbatmop/MNBVC
4,123Vezi pe GitHubMNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS
Massive, diverse Chinese text corpus from internet sources.
chinesechinese-languagechinese-nlp - luodian/otter
Luodian/Otter
3,410Vezi pe GitHub🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
Multimodal in-context instruction tuning data.
Python - opengvlab/internvideo
OpenGVLab/InternVideo
2,292Vezi pe GitHubECCV2024 Video Foundation Models & Data for Multimodal Understanding
Video-centric instruction dataset for chat-based understanding.
Pythonaction-recognitionbenchmarkcontrastive-learning - microsoft/llava-med
microsoft/LLaVA-Med
2,214Vezi pe GitHubLarge Language-and-Vision Assistant for Biomedicine, built towards multimodal GPT-4 level capabilities.
Biomedical instruction-following dataset for vision-language models.
Python - openmotionlab/motiongpt
OpenMotionLab/MotionGPT
1,938Vezi pe GitHubNeurIPS 2023 MotionGPT: Human Motion as a Foreign Language, a unified motion-language generation model using LLMs
Human motion-related instruction tuning tasks.
Python3d-generationchatgptgpt - lyuchenyang/macaw-llm
lyuchenyang/Macaw-LLM
1,590Vezi pe GitHubMacaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration
Multi-turn dialogue dataset for multimodal integration.
Pythondeep-learninglanguage-modelmachine-learning - mbzuai-oryx/video-chatgpt
mbzuai-oryx/Video-ChatGPT
1,504Vezi pe GitHubACL 2024 🔥 Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
High-quality video instruction dataset for detailed understanding.
Pythonchatbotclipgpt-4 - kakaobrain/coyo-dataset
kakaobrain/coyo-dataset
1,256Vezi pe GitHubCOYO-700M: Large-scale Image-Text Pair Dataset
Large-scale image-text pairs for pre-training.
Python - cluebenchmark/cluecorpus2020
CLUEbenchmark/CLUECorpus2020
1,012Vezi pe GitHubLarge-scale Pre-training Corpus for Chinese 100G 中文预训练语料
Cleaned 100GB Chinese corpus for pre-training and NLP tasks.
albertbertchinese - plexpt/chatgpt-corpus
PlexPt/chatgpt-corpus
This project provides a comprehensive Chinese language corpus designed to support the training and fine-tuning of large language models. It serves as a structured natural language processing resource, offering a collection of text data that includes dialogue, customer service interactions, and creative writing. The dataset is organized into distinct thematic categories, allowing for targeted model development across specific conversational and narrative contexts. By providing information in standardized, schema-agnostic text formats, the collection ensures portability across various machine l
Provides a large-scale Chinese language text collection including dialogue and fiction for training large language models.
awesomecorpuscorpus-data - optimalscale/detgpt
OptimalScale/DetGPT
2023-06-13 Added tuned linear weights for Vicuna-7b. 2023-05-25 Our paper is available at this link. 2023-05-09 We have launched our project website. 2023-05-08 The first version of DetGPT is available now! Try our demo.
Query-answer pairs for visual detection and reasoning.
Jupyter Notebook - stevengrove/gpt4tools
StevenGrove/GPT4Tools
GPT4Tools is an intelligent system that can automatically decide, control, and utilize different visual foundation models, allowing the user to interact with images during a conversation.
Tool-use instruction datasets for language models.
Python - phellonchen/x-llm
phellonchen/X-LLM
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
Chinese multimodal instruction dataset for foreign language treatment.
Python - openlamm/lamm
OpenLAMM/LAMM
NeurIPS 2023 Datasets and Benchmarks Track LAMM: Multi-Modal Large Language Models and Applications as AI Agents
Comprehensive dataset for language-assisted multimodal tuning.
Python - fuxiaoliu/lrv-instruction
FuxiaoLiu/LRV-Instruction
ICLR'24 Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Robust instruction tuning to mitigate model hallucinations.
Pythonchatgptevaluationevaluation-metrics - mobvoi/seq-monkey-data
mobvoi/seq-monkey-data
Large-scale dataset used for training general-purpose language models.
- vt-nlp/multiinstruct
VT-NLP/MultiInstruct
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Benchmark dataset for multimodal zero-shot learning.
Python - icoz69/stablellava
icoz69/StableLLAVA
Official repo for StableLLAVA
Synthesized image-dialogue data for instruction tuning.
Python - polyu-chenlab/etbench
PolyU-ChenLab/ETBench
👾 E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024)
Time-sensitive video understanding instruction dataset.
Python
