# jadianes/spark-py-notebooks **Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/jadianes-spark-py-notebooks).** _How this analysis was created: the description and tags below were written by an AI model that read this project's README and public documentation pages; stars, license and language come straight from the GitHub API. The model does not read the source code._ 1,661 stars · 906 forks · Jupyter Notebook · NOASSERTION ## Links - GitHub: https://github.com/jadianes/spark-py-notebooks - Homepage: http://jadianes.github.io/spark-py-notebooks - awesome-repositories: https://awesome-repositories.com/repository/jadianes-spark-py-notebooks.md ## Topics `big-data` `bigdata` `data-analysis` `data-science` `ipython` `ipython-notebook` `machine-learning` `mllib` `notebook` `pyspark` `python` `spark` ## Description This repository serves as an educational collection of Jupyter notebooks designed to demonstrate distributed data processing and machine learning workflows. It provides a structured resource for learning how to perform large-scale statistical analysis, execute relational queries, and develop predictive models using Python and Apache Spark. The project distinguishes itself by offering practical, interactive guides that bridge the gap between theoretical distributed computing concepts and applied data science. By utilizing notebook environments, it enables users to document and execute code for exploratory data analysis, model training, and data transformation in a reproducible research format. The collection covers a broad capability surface, including distributed computing primitives for aggregating and filtering massive datasets, as well as algorithmic implementations for classification and regression tasks. It also addresses the deployment of trained models as web-accessible services for real-time prediction. The repository is organized as a series of tutorials and code examples, providing a hands-on environment for users to explore big data analytics and machine learning techniques. ## Tags ### Data & Databases - [Apache Spark Pipelines](https://awesome-repositories.com/f/data-databases/apache-spark-pipelines.md) — Provides development and execution environments for large-scale data processing using Apache Spark. - [Distributed Data Processing Frameworks](https://awesome-repositories.com/f/data-databases/distributed-data-processing-frameworks.md) — Provides systems for partitioning, transforming, and processing large-scale datasets across distributed computing clusters. ([source](https://github.com/jadianes/spark-py-notebooks#readme)) - [Distributed Datasets](https://awesome-repositories.com/f/data-databases/distributed-datasets.md) — Manages distributed data collections supporting lazy transformations and parallel processing. - [Dataframe Processing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing/dataframe-processing.md) — Enables programmatic manipulation of tabular datasets for statistical and machine learning workflows. ([source](https://github.com/jadianes/spark-py-notebooks/tree/master/nb10-sql-dataframes)) - [In-Memory Caching](https://awesome-repositories.com/f/data-databases/dataset-management-tools/in-memory-caching.md) — Provides tools for caching large datasets in memory to improve application performance. - [Exploratory Data Analysis](https://awesome-repositories.com/f/data-databases/exploratory-data-analysis.md) — Facilitates the process of cleaning and manipulating datasets to discover patterns and statistical insights. - [Lazy Query Execution](https://awesome-repositories.com/f/data-databases/lazy-query-execution.md) — Utilizes deferred computation patterns for efficient data transformation. - [SQL Query Execution](https://awesome-repositories.com/f/data-databases/sql-query-execution.md) — Executes SQL statements and retrieves results as structured data. ([source](https://github.com/jadianes/spark-py-notebooks/tree/master/nb10-sql-dataframes)) - [Tabular Data Analysis](https://awesome-repositories.com/f/data-databases/tabular-data-analysis.md) — Supports analytical processing of data organized in tables, including cleaning, pivoting, and feature engineering. ([source](https://github.com/jadianes/spark-py-notebooks#readme)) ### Artificial Intelligence & ML - [Distributed Learning](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/distributed-and-scaling-strategies/distributed-learning.md) — Provides techniques and frameworks for training machine learning models across multiple computing nodes. - [Exploratory Data Analysis](https://awesome-repositories.com/f/artificial-intelligence-ml/data-preparation/exploratory-data-analysis.md) — Provides tools for loading, cleaning, visualizing, and summarizing datasets to understand their structure. ([source](https://github.com/jadianes/spark-py-notebooks#readme)) - [Decision Tree Induction](https://awesome-repositories.com/f/artificial-intelligence-ml/decision-trees/decision-tree-induction.md) — Implements decision tree training algorithms for constructing predictive models on large datasets. ([source](https://github.com/jadianes/spark-py-notebooks/tree/master/nb9-mllib-trees)) - [Logistic Regression Models](https://awesome-repositories.com/f/artificial-intelligence-ml/logistic-regression-models.md) — Provides distributed logistic regression training for building binary classification models. ([source](https://github.com/jadianes/spark-py-notebooks/tree/master/nb8-mllib-logit)) - [Distributed Training Guides](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning-guides/distributed-training-guides.md) — Provides educational resources focusing on scaling model training across multiple compute nodes. - [Model Training](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training.md) — Provides distributed algorithms and procedures for training predictive classification and regression models on historical data. ([source](https://github.com/jadianes/spark-py-notebooks/blob/master/README.md)) ### Part of an Awesome List - [Jupyter Notebook Collections](https://awesome-repositories.com/f/awesome-lists/learning/jupyter-notebook-collections.md) — Provides a collection of interactive notebooks for learning and executing data science and machine learning workflows. - [Data Engineering and Analytics](https://awesome-repositories.com/f/awesome-lists/data/data-engineering-and-analytics.md) — Offers tools for big data processing, streaming, and database management. ### Development Tools & Productivity - [Notebook Execution Environments](https://awesome-repositories.com/f/development-tools-productivity/code-execution-environments/notebook-execution-environments.md) — Delivers interactive environments that allow for the execution of code snippets directly within notebook interfaces. - [Interactive Data Science Environments](https://awesome-repositories.com/f/development-tools-productivity/interactive-data-science-environments.md) — Offers browser-based platforms for iterative analysis combining code, visualization, and narrative. ### Scientific & Mathematical Computing - [Statistical Analysis Libraries](https://awesome-repositories.com/f/scientific-mathematical-computing/numerical-mathematical-foundations/statistics-probability/statistical-analysis-libraries/statistical-metric-calculators/statistical-analysis-libraries.md) — Provides comprehensive toolsets for calculating descriptive statistics and correlations across datasets. ([source](https://github.com/jadianes/spark-py-notebooks/blob/master/README.md)) ### Software Engineering & Architecture - [Directed Acyclic Graph Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/parallel-processing-pipelines/directed-acyclic-graph-pipelines.md) — Implements task execution systems that use directed acyclic graphs to manage sequential and parallel processing dependencies.