Menu Close

Johannesburg: Senior Data Engineer (Spark & Python Specialist) posted by Hire Resolve

Job Description

An award-winning privacy-preserving data collaboration platform that enables companies to analyze and collaborate on consumer data to gain insights, build predictive models, and monetize data without sharing the raw data or compromising consumer privacy, is seeking a Senior Cloud Data Engineer who will be a lead technical contributor responsible for building and optimizing high-performance data processing engines. 

Responsibilities:

  • Spark Optimization: Act as the internal SME for Spark internals; manage memory, shuffle tuning, and partitioning for cost-effective performance.

  • Cloud-Agnostic Development: Build pipelines using Python and Delta Lake, decoupling code from specific cloud providers and reducing reliance on GUI tools (e.g., ADF).

  • Refactoring & Modernization: Migrate complex SQL-based ETL into modular, testable, and maintainable Python libraries.

  • Lakehouse Engineering: Manage Medallion Architecture (Bronze/Silver/Gold) using Delta Lake, ensuring storage performance via Z-Ordering and Vacuuming.

  • Code-First Orchestration: Support the transition to code-centric patterns (Airflow, Dagster) to prioritize portability.

  • Technical Excellence: Lead code reviews, mentor junior engineers, and implement automated testing frameworks (Pytest).


Minimum Requirements:

  • Education: Bachelors degree in Computer Science, Information Systems, Engineering, or a related field.
  • Spark Mastery: 6+ years of Spark/PySpark experience; expert ability to diagnose bottlenecks via Spark UI and optimize complex DAGs.

  • Advanced Python: Proficiency in production-grade Python, including building reusable libraries and automated testing.

  • Azure Ecosystem: Strong experience with Azure Synapse, Dedicated SQL Pools, and Data Factory.

  • Modern Data Stack: Hands-on experience with Delta Lake, Parquet, and containerization (Docker).

  • Migration Skills: Solid T-SQL skills to interpret and migrate legacy logic into Python-centric environments.

  • Security & Governance: Proven ability to implement high levels of security and compliance across data processes.

Benefits:

  • Competitive salary based on experience (salary can potentially be more based on experience/skills)

IF you meet the above requirements and want to make a career-changing move, apply today by emailing your CV to .za

How to Apply

Click the green “Go Apply” button below to apply directly online with the employer.

About Other IT/Computer Jobs in Gauteng

The IT and computer industry is one of the most dynamic and growing sectors in Gauteng, with a thriving job market that offers a wide range of opportunities for career advancement.

In terms of salary ranges, IT professionals in Gauteng can expect to earn anywhere from R800 000 per annum for junior positions to over R2 million for senior executives. The average salary for software engineers is around R1.5 million, while data analysts can earn between R900 000 and R1.2 million. With experience and skills, IT professionals in Gauteng can expect significant salary increases, with some roles offering bonuses and benefits that can increase their overall remuneration.

To succeed in the IT industry in Gauteng, candidates need to possess a range of key skills, including programming languages such as Java, Python, and C++, as well as experience with databases, cloud computing, and cybersecurity. Strong problem-solving skills, attention to detail, and excellent communication skills are also essential for success in this field. Additionally, knowledge of ITIL (Information Technology Infrastructure Library) and Agile methodologies can be highly valued by employers.

Several major companies and industries in Gauteng are currently hiring IT professionals, including technology firms such as Microsoft, IBM, and Oracle, as well as banks and financial institutions like Standard Bank and Nedbank. The finance industry is also a significant employer of IT staff, with roles ranging from software development to data analytics and cybersecurity.

Career growth opportunities are abundant in the IT industry in Gauteng, with many companies investing heavily in employee training and development programs. With experience and skills, IT professionals can move into senior leadership positions or start their own businesses, while also pursuing further education and certifications. The Johannesburg Stock Exchange (JSE) is also a significant employer of IT staff, with roles ranging from software development to data analytics and cybersecurity.

Overall, the IT industry in Gauteng offers a wide range of exciting career opportunities for job seekers with the right skills and experience. With salaries that are competitive with those offered by major companies globally, there has never been a better time to pursue a career in this dynamic and rapidly evolving field.


This information provides general career guidance. Actual salaries and requirements vary by employer.



GO APPLY NOW

Safe & secure application process


Get Job Alerts

Never miss a job! Get daily alerts matching your preferences.

📧 Why Subscribe?

  • Get notified instantly when new jobs match your criteria
  • No registration required - just your email
  • Daily digest of relevant opportunities
  • Free forever - never miss your dream job
  • Unsubscribe anytime with one click
We'll never share your email with anyone
Leave blank for all locations
We'll send you a daily summary of new matching jobs
🔒 Privacy Guaranteed
Your email is safe with us. We only send job alerts and never spam. Unsubscribe anytime with one click. No registration, no account required.


Jobseeker Tip

Apply early, keep your CV ready, and search using both job title and location. Example: Admin Johannesburg.

Interview tips

How to Apply

Click “GO APPLY NOW” to visit the company’s application page.
Follow their instructions carefully.

JVR Jobs connects you with employers – we don’t process applications directly.

Browse Employers

View All

Job Alerts


Share this to someone who needs a job:
Posted in Jobs in Central Johannesburg, Jobs in Gauteng, Jobs in Johannesburg, Jobs in South Africa

More Jobs in Your Area