1 repository
Awesome GitHub RepositoriesParallel Text Alignments
Algorithms that match corresponding text segments across different languages to create paired datasets for translation and LLM training.
Distinct from Acoustic-Text Alignment: None of the candidates cover the linguistic alignment of parallel corpora; they focus on UI or audio alignment.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Parallel Text Alignments. Refine with filters or upvote what's useful.
Awesome Parallel Text Alignments GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- esbatmop/mnbvc
esbatmop/MNBVC
4,123Vezi pe GitHubMNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS
Implements specialized alignment algorithms to create paired multilingual training datasets.
chinesechinese-languagechinese-nlp
