8 Repos
Utilities for downloading, organizing, and converting public datasets.
Distinguishing note: Focuses on the end-to-end preparation of specific public datasets.
Explore 8 awesome GitHub repositories matching data & databases · Dataset Preparation Scripts. Refine with filters or upvote what's useful.
This project is a modular research toolkit designed for developing, training, and evaluating deep learning models for object detection, segmentation, and video instance tracking. It provides a flexible training engine that manages complex neural network execution, including distributed training, custom lifecycle hooks, and weight optimization. The framework is built around a hierarchical configuration system that allows users to define architectures, data pipelines, and training hyperparameters through composable, inheritable files. The project distinguishes itself through its highly modular
Provides utilities to prepare and convert ADE20K datasets for training.
LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training and evaluating models that integrate visual, textual, and audio data, serving as a cross-modal feature extractor and a zero-shot visual reasoning engine. The framework distinguishes itself by using frozen-backbone integration, where pretrained encoders remain non-trainable while lightweight adapter layers are updated. It employs cross-modal feature alignment to map different representations into a shared embedding space and utilizes a modular model wrapper to swap vision and
Automates the downloading and organization of large collections of language-vision datasets and their annotations.
This project is a neural machine translation system used to build models that automatically translate text from one language to another. It utilizes sequence-to-sequence modeling to transform variable-length input sequences into corresponding output sequences. The system implements bidirectional recurrent neural network encoding and attention mechanisms to capture contextual information and focus on specific parts of the source text during translation. To manage training and inference, it employs separate computational graphs and supports distributing model layers across multiple GPU devices.
Constructs input pipelines that clean and zip source-target translation pairs with sequence padding.
MMDetection3D is an open-source toolbox for 3D perception, providing a unified framework for detecting and segmenting objects in three-dimensional environments. It supports a range of core tasks including monocular 3D object detection from single camera images, LiDAR-based 3D object detection from raw point clouds, and multi-modal fusion that combines camera images with LiDAR data. The toolbox also covers point cloud semantic segmentation, assigning class labels to every point in a scan for scene understanding. The project distinguishes itself through a config-driven pipeline that orchestrate
Downloads, organizes, and preprocesses supported 3D datasets into expected folder structures and annotation formats.
This repository provides a complete framework for training generative adversarial networks (GANs) that produce high-resolution photorealistic images, up to 1024 by 1024 pixels. The core technique is progressive layer growth, where both the generator and discriminator networks start training at low resolution and gradually add new layers to model finer details, enabling stable synthesis of large images. The framework includes a high-resolution image generator, an image quality metric evaluator, a latent space interpolation tool for creating smooth transition videos, and a multi-resolution datas
Converts image datasets into multi-resolution TFRecords for efficient streaming during progressive training.
Dieses Projekt ist eine Sammlung von Big-Data-Frameworks und Pipelines, darunter ein Apache Hive-Analyse-Framework, eine Plattform für Verhaltensdatenanalyse, eine Predictive-Analytics-Engine und Echtzeit-Datenpipelines. Es bietet die Infrastruktur für den Aufbau von ETL-Workflows (Extract, Transform, Load), um große Datensätze für verteilte Speicherung und SQL-basierte Analysen zu verarbeiten. Das System unterstützt diverse analytische Implementierungen, wie eine Predictive-Engine mittels linearer Regression für Prognosen und eine Echtzeit-Architektur, die Daten über Message-Broker für sofortiges Reporting weiterleitet. Es enthält spezialisierte Funktionen für die Analyse von Nutzerverhalten, E-Commerce-Performance-Messungen und Daten des städtischen Nahverkehrs. Die Codebasis deckt ein breites Spektrum an Data Engineering und Analyse ab, einschließlich Datenbereinigung und -transformation, verteilter Datenaufnahme (Ingestion), fensterbasierter Stream-Verarbeitung und der Visualisierung von Ergebnissen durch Business-Intelligence-Tools. Zudem ermöglicht es die Berechnung spezifischer Geschäftskennzahlen wie Konversionsraten, Monetarisierungs-Performance und Nutzer-Engagement-Level.
Includes processes to merge data files and filter fields to optimize memory usage before loading into databases.
OpenPrompt ist ein Prompt-Learning-Framework, das darauf ausgelegt ist, Large Language Models an nachgelagerte NLP-Aufgaben anzupassen. Es bietet ein umfassendes Toolkit zur Implementierung von manuellen, Soft- und kontinuierlichen Prompting-Strategien, wodurch Modelle verfeinert werden können, ohne alle zugrunde liegenden Parameter aktualisieren zu müssen. Das Projekt zeichnet sich durch die Unterstützung von diskretem und kontinuierlichem Prompt-Tuning aus. Es enthält ein System zum Injizieren trainierbarer Soft-Tokens und Embeddings in Modelleingaben mittels Gradientenabstieg sowie eine automatische Prompt-Generierungs-Engine, die Beam-Search und generative Modelle verwendet, um hochwahrscheinliche Text-Templates für spezifische Datensätze zu finden. Das Framework deckt mehrere Kernbereiche ab, darunter Template-Design und Label-Verbalisierung zur Zuordnung von Klassifizierungs-Labels zu Vokabelwörtern. Es bietet zudem Modell-Anpassungstools zum Umhüllen vortrainierter Modelle, Logit-Kalibrierung zur Verbesserung der Vorhersagegenauigkeit sowie eine Daten-Pipeline mit spezialisierter Sampling-Logik für Few-Shot-Learning. Trainings- und Experiment-Workflows werden über Konfigurationsdateien verwaltet, die Lernszenarien, Hyperparameter und Pipeline-Spezifikationen definieren.
Processes raw data into standardized input examples and labels using pre-defined data processors.
Map-anything is a 3D scene reconstruction framework and neural geometry estimator designed to transform two-dimensional images into metric three-dimensional spatial representations using feed-forward neural networks. It provides a specialized toolkit for predicting camera intrinsics and ray directions from single images without requiring external geometric metadata. The project includes a 3D model benchmarking suite that utilizes a unified model wrapper to standardize outputs from diverse reconstruction models. This allows for consistent evaluation and accuracy measurement across various spat
Provides scripts for loading and organizing images from synthetic and real-world spatial datasets for model training.