Johannesburg: Databricks Data Engineer (PySpark, Financial Datasets) – Johannesburg – up to R500 per hour posted by E-Merge

About the Role

We are seeking a skilled Databricks Data Engineer to join our team in Johannesburg, South Africa. As a forward-thinking technology company operating across the Middle East, Asia Pacific, Africa, and the United States, we are dedicated to advancing cloud innovation and developing AI solutions that positively impact society. In this role, you will collaborate with a team delivering sophisticated AI, data, and cloud capabilities that support the banking and financial sectors. You will utilize Databricks to design scalable data pipelines, enhance analytical performance, and generate actionable insights that accelerate digital transformation.

Key Responsibilities

• ETL/ELT Development: Develop, test, and deploy robust and efficient data pipelines using PySpark/Scala and the Databricks platform (including Delta Lake and Databricks Workflows).
• Data Transformation: Implement complex data transformation logic to clean, enrich, and aggregate financial data from various source systems (e.g., core banking, trading platforms).
• Cloud Integration: Integrate Databricks with native cloud services (AWS, Azure, or GCP) for data ingestion (e.g., S3, ADLS) and workflow orchestration (e.g., Azure Data Factory, AWS Glue).
• Quality and Testing: Write unit and integration tests for data pipelines and apply data quality checks to ensure accuracy in financial reporting and analysis.
• Compliance Support: Apply basic security and access control policies, such as those governed by Unity Catalog, to adhere to the firm’s compliance requirements.
• Performance: Assist in monitoring and tuning Databricks cluster configurations and Spark job parameters to improve efficiency and reduce cost.

Requirements

• Strong hands-on experience with Databricks notebooks, clusters, job scheduling, and understanding of the Delta Lake transaction log for reliability.
• Proficient in Python (especially PySpark) and expert in SQL.
• Solid understanding of Apache Spark fundamentals (e.g., RDDs, DataFrames, lazy evaluation) and distributed computing.
• Practical experience with dimensional modeling (star/snowflake schemas) and implementing the Medallion Architecture (Bronze, Silver, Gold layers) on Delta Lake.
• Familiarity with version control (Git) and experience with basic CI/CD processes for deploying Databricks code.

Qualifications

• Bachelor’s degree in Computer Science or related field.

Salary & Benefits

Up to R500 per hour salary negotiable based on experience.