palantir/pyspark-style-guide
Pyspark Style Guide
The project establishes consistent naming conventions, readability standards, and structural best practices across large PySpark data transformation codebases. It provides architectural patterns and performance recommendations designed to prevent common data transformation pitfalls in distributed systems, alongside recommendations for organizing code logic and structuring expressions within large-scale data processing codebases.
The guidelines cover performance optimization through the replacement of custom user-defined functions with native operations, big data join safety by validating key uniqueness and declaring explicit join types, and data processing practices such as explicit schema enforcement, deterministic window framing, and safe handling of missing values. Additional focus areas include expression chaining decomposition and rule-based syntax analysis to flag non-idiomatic patterns.
Features
- Code Style Guides - Establishes consistent naming conventions, readability standards, and structural best practices across large data transformation codebases.
- Explicit Row Frame Configurations - Defines exact row or range frames for window operations to ensure predictable execution behavior.
- Explicit Selection Boundary Schemas - Defines expected schemas at transform boundaries using explicit select operations rather than dropping columns later.
- Early Selection Schema Contracts - Specifies expected input and output data structures early using explicit selection statements.
- Cartesian Join Prevention - Validates key uniqueness and join types to prevent unintended Cartesian products and row explosions during distributed merges.
- Null Value Handling - Handles missing values safely by utilizing null indicators instead of magic strings to preserve native validation.
- Data Pipeline Optimizations - Improves execution throughput in distributed data processing pipelines by leveraging fast native operations.
- Table Joining Operations - Manages table joins safely by verifying key uniqueness and resolving column name collisions beforehand.
- User-Defined Functions - Replaces slow user-defined functions with native data operations to eliminate execution bottlenecks in distributed systems.
- Native Performance Optimizations - Replaces slow custom procedural routines with built-in native operations to accelerate distributed data processing.
- Deterministic Window Frame Bounds - Enforces explicit row and range boundaries on analytical window operations to ensure predictable distributed calculations.
- Consistent Window Calculation Frames - Configures explicit row or range frames for window calculations to ensure consistent execution results across environments.
- Transformation Boundary Schema Contracts - Enforces strict data structure contracts at transformation boundaries to clarify expectations for downstream processing.
- Functional Declarative Patterning - Replaces imperative row loops with declarative functional compositions to optimize distributed execution performance.
- Chaining Length Limits - Restricts nested operation lengths inside single statements to improve readability and encourage helper function extraction.
- Empty String to Null Converters - Populates empty dataset columns with standard null values rather than sentinel strings to preserve semantic correctness.
- Convention Rule AST Analyzers - Analyzes abstract syntax trees against defined coding conventions to flag non-idiomatic data frame operations.
- SIMD Vectorised Kernels - Bypasses row-by-row interpreter bottlenecks by replacing custom procedural routines with vectorised native operations.
- Engineering Best Practices - Provides architectural patterns and performance recommendations to prevent common data transformation pitfalls in distributed systems.
- Python Coding Standards - Establishes coding standards for organizing logic, naming variables, and structuring expressions within Python data pipelines.
