South Africa: Senior Data Engineer (Spark & Python Specialist) posted by Recruit-It

About the Role

Our client is seeking a Senior Cloud Data Engineer to join their Engineering team at the company. In this role, you will be a key technical contributor responsible for building, optimizing, and maintaining high-performance data processing engines.

Key Responsibilities

• Spark Implementation & Optimization: Act as a senior technical resource for Spark internals, applying best practices in memory management, shuffle tuning, and partitioning to ensure their Spark-based processing is performant and cost-effective.
• Cloud-Agnostic Development: Develop and maintain data pipelines using Python, PySpark, and Delta Lake/Parquet, adhering to their strategy of decoupling code from specific cloud provider services and reducing reliance on GUI-based tools like ADF.
• Refactoring & Modernization: Contribute to the ongoing evolution of the companies platform by refactoring complex SQL-based ETL into modular, testable, and maintainable Python libraries.
• Lakehouse Engineering: Build and optimize Medallion Architecture (Bronze/Silver/Gold) layers using Delta Lake, ensuring efficient data versioning, schema evolution, and storage performance (Z-Ordering, Vacuuming).
• Code-First Orchestration: Support the transition toward code-centric orchestration patterns (e.g., Airflow, Dagster, or Python-based wrappers) that prioritize portability and reduce dependency on cloud-specific orchestration tools.
• Technical Excellence: Participate in code reviews, mentor junior engineers in PySpark best practices, and contribute to the development of automated testing frameworks (Pytest) to ensure high-quality, reliable data delivery.

Requirements

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics & Dedicated SQL Pools: Solid experience working within Azure Synapse, specifically with Dedicate SQL Pools and Data Factory working with complex pipelines.
• Data Lakehouse Expertise: Hands-on experience with Delta Lake and Parquet, specifically in high-volume environments.
• Agnostic Mindset: Experience working with open-source standards and containerization (Docker) to ensure data workloads can be ported across different cloud infrastructures.
• Collaborative Engineering: Proven track record of working within a team to successfully deliver and modernize large-scale data workloads.

Qualifications

Bachelor’s degree in computer science, Information Systems, Engineering, or a closely related field.