South Africa: Senior Data Engineer (Spark & Python Specialist) posted by Recruit-It

Role:
Our client is seeking a Senior Cloud Data Engineer to join their Engineering team at the company. In this role, you will be a key technical contributor responsible for building, optimizing, and maintaining high-performance data processing engines.
 
You will apply your deep expertise in Spark Best Practices to develop modular, portable PySpark applications, ensuring the companies platform remains provider-agnostic while operating within modern environments like Microsoft Fabric.
 
As an experienced member of the Data Engineering team, you will focus on refactoring legacy logic into scalable Python-centric solutions, optimizing the companies’ data lakehouse architecture, and contributing to the overall technical excellence of their data platform.
 
Key Responsibilities

• Spark Implementation & Optimization: Act as a senior technical resource for Spark internals, applying best practices in memory management, shuffle tuning, and partitioning to ensure their Spark-based processing is performant and cost-effective.
• Cloud-Agnostic Development: Develop and maintain data pipelines using Python, PySpark, and Delta Lake/Parquet, adhering to their strategy of decoupling code from specific cloud provider services and reducing reliance on GUI-based tools like ADF.
• Refactoring & Modernization: Contribute to the ongoing evolution of the companies platform by refactoring complex SQL-based ETL into modular, testable, and maintainable Python libraries.
• Lakehouse Engineering: Build and optimize Medallion Architecture (Bronze/Silver/Gold) layers using Delta Lake, ensuring efficient data versioning, schema evolution, and storage performance (Z-Ordering, Vacuuming).
• Code-First Orchestration: Support the transition toward code-centric orchestration patterns (e.g., Airflow, Dagster, or Python-based wrappers) that prioritize portability and reduce dependency on cloud-specific orchestration tools.
• Technical Excellence: Participate in code reviews, mentor junior engineers in PySpark best practices, and contribute to the development of automated testing frameworks (Pytest) to ensure high-quality, reliable data delivery.

 
Required Skills and Experience

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics & Dedicated SQL Pools: Solid experience working within Azure Synapse, specifically with Dedicate SQL Pools and Data Factory working with complex pipelines
• Data Lakehouse Expertise: Hands-on experience with Delta Lake and Parquet, specifically in high-volume environments.
• Agnostic Mindset: Experience working with open-source standards and containerization (Docker) to ensure data workloads can be ported across different cloud infrastructures.
• Collaborative Engineering: Proven track record of working within a team to successfully deliver and modernize large-scale data workloads.
• Ensure high levels of security, compliance, and data governance across data engineering processes.

 
Collaboration & Leadership

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
• Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
• Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
• Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.

 
Required Qualifications

• Bachelor’s degree in computer science, Information Systems, Engineering, or a closely related field.

 
Why you’ll love working for the company
They believe in taking care of their team and creating an environment where you can thrive. As part of their company, you’ll enjoy:
 

• Flexible Working Arrangements: Whether you’re a night owl or an early bird, the company offers hybrid and remote options to suit your lifestyle
• Comprehensive Benefits: From a wellness program to home office reimbursements and continuous learning opportunities, the company has got you covered.
• Team Culture: Fun team-building activities, regular socials, and a supportive, inclusive culture that values transparency, accountability, and work-life balance.
• Performance Incentives: Competitive salaries, ESOP, and recognition for your hard work.