3 个仓库
Awesome GitHub RepositoriesDistributed Dataframes
Dataframe abstractions that distribute computation across multiple nodes or cores to handle large-scale datasets.
Distinct from Dataframe Processing: Distinct from general Dataframe Processing: specifically focuses on the distributed orchestration of tabular data across clusters.
Explore 3 awesome GitHub repositories matching data & databases · Distributed Dataframes. Refine with filters or upvote what's useful.
Awesome Distributed Dataframes GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- eventual-inc/daft
Eventual-Inc/Daft
5,225在 GitHub 上查看Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho
Provides a distributed dataframe library for processing large-scale structured and unstructured data across local cores or Kubernetes clusters.
Rustai-engineeringai-pipelinearrow - johnsnowlabs/spark-nlp
JohnSnowLabs/spark-nlp
4,135在 GitHub 上查看Spark NLP 是一个构建在 Apache Spark 分布式计算框架之上的可扩展文本分析和机器学习工具包。它提供了一个多模态机器学习框架和一个用于对标注器进行排序以处理大规模语言数据的分布式流水线系统。该库包含一个用于生成上下文向量嵌入的 Transformer 文本处理器，以及一个用于管理大型语言模型的专用推理引擎。 该项目通过其在统一视觉-语言架构内处理异构数据类型（包括文本、音频和图像）的能力而脱颖而出。它支持高级生成式 AI 功能，如提示工程、具有约束 JSON 输出的结构化实体提取，以及消除网络延迟的本地推理。此外，它还提供跨文本和图像模态的跨语言翻译和零样本分类工具。 该框架涵盖了广泛的功能，包括用于实体识别和情感分析的监督模型训练，以及抽取式问答和文档摘要。它集成了向量数据库支持以进行相似性搜索，并为 GPU 加速和通过集中式注册表进行模型生命周期管理提供了基础设施。 该工具包允许通过公共仓库分发自定义模型和流水线，并支持通过 REST API 部署模型。
Leverages distributed computing clusters to execute natural language processing tasks across massive datasets using distributed dataframes.
Scala - databricks/spark-the-definitive-guide
databricks/Spark-The-Definitive-Guide
3,099在 GitHub 上查看This project is an educational resource and technical manual for Apache Spark, focused on the architecture and practical application of large-scale data processing. It serves as a guide for big data engineering and distributed computing, covering the principles of parallel processing and fault-tolerant data distribution. The material provides instructional content on designing distributed ETL pipelines and implementing data analysis workflows. It includes tutorials for polyglot data processing, offering patterns and examples for using Python, Scala, and Java within a unified environment. The
Explains the abstraction of distributed dataframes for parallel processing across compute clusters.
Scala
