Menu Close

Johannesburg: Big Data Data Engineer

Johannesburg: Big Data Data Engineer

Posted on 2025-09-04 00:00:00

Employer Unspecified
Category IT / Computing / Software
Location Gauteng  / Johannesburg

Job Summary

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers. Key Responsibilities: Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS). Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL. Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics. Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process. Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready. Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment. Maintain metadata, lineage, and documentation for transparency and governance. Monitor pipeline performance and implement error handling and alerting mechanisms. Technical Skills & Experience: 3 years of experience in data engineering or ETL development within a big data environment. Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop. Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language). Experience working with unstructured JSON data and transforming it into structured formats. Solid understanding of data lake architectures: Raw, Published, and Functional layers. Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi. Experience with schema design, data modeling, and partitioning strategies. Comfortable with version control tools (e.g., Git) and CI/CD processes. Nice to Have: Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation). Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc. Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment. Familiarity with data quality frameworks (e.g., Deequ, Great Expectations). Qualifications: Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field. Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus. * In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent * If you have not received any feedback after 2 weeks, please consider you application as unsuccessful.

View Job  Franschhoek: Waiter posted by Craven Cottage CC

Click Go Apply to apply online!

Get Similar Job Alerts
Enter your WhatsApp number (e.g., +27740908132)
You'll receive alerts for new jobs in Johannesburg, Gauteng


More jobs in Johannesburg




More jobs in Gauteng




Browse IT / Computing / Software Jobs




Browse Unspecified Jobs



Apply directly for this position. Please read all instructions carefully.

We do not process job applications; we simply aggregate and display job listings.

More related positions

Johannesburg: Content Creator posted by Liham Consulting Pty (Ltd)

Job DescriptionPreparing print-ready designs, including formatting, resizing, and image adjustments while ensuring brand and colour consistency.Creating...


View Job
Content Creator

Johannesburg: Supply Chain Coordinator posted by Headhunters

Our client, a leader in the automotive component manufacturing sector, is currently looking to employ a Supply Chain Coordinator based in Elandsfontein,...


View Job
Supply Chain Coordinator

Johannesburg: Project Manager posted by Daisy JHB (Pty) Ltd

Purpose of the roleThe Project Manager is responsible for planning, organizing, and executing projects, ensuring they are completed on time, within budget,...


View Job
Project Manager

Johannesburg: Customer Service Representative – Mbombela posted by Tailormade Solution

Drive sales efforts through effective planning, scheduling and conducting ongoing customer visitsEngage with existing customer base in order to grow...


View Job
Customer Service Representative – Mbombela

Johannesburg: Part Sales Executive – Wanted immediately posted by First degree Recruitment

We are seeking a dynamic and customer-focused Part Sales Executive to join our team. In this role, you will be responsible for selling automotive parts to...

View Job  Johannesburg: Finance Manager

View Job
Part Sales Executive – Wanted immediately

Johannesburg: Exports Logistics Coordinator – Johannesburg posted by Hire Resolve

A leading manufacturer and distributor in the FMCG sector is seeking an experienced Exports Logistics Coordinator to join their fast-paced team based in...


View Job
Exports Logistics Coordinator – Johannesburg

Johannesburg South: PROPERTY ADMINISTRATOR – REQUIRED URGENTLY posted by ERZ Telecoms

- Debtors Management Administration- Creditors Management Administration- Assisting with Drawing up of leases- Rent Roll-MDA experience will be...


View Job
PROPERTY ADMINISTRATOR – REQUIRED URGENTLY

Johannesburg: Oracle Finance Functional Consultant posted by Hire Resolve

? Oracle Finance Functional Consultantnbsp ? LocationnbspJohannesburg ? Rate Competitive Based on experience? Start Immediate NegotiableA leading...


View Job
Oracle Finance Functional Consultant

Johannesburg: Maintenance Manager – Johannesburg posted by Hire Resolve

A leading manufacturer in the plastics and packaging industry is seeking an experienced Maintenance Manager with strong mechanical and electrical expertise...


View Job
Maintenance Manager – Johannesburg

Johannesburg: Mid-Level Software Developers posted by Hire Resolve

An established PR and media company based in Johannesburg is seeking the expertise of two Mid-Level Software Developers to join their growing remote...


View Job
Mid-Level Software Developers

Johannesburg: Promotions and Events Manager posted by Edgexec (Pty) Ltd

Promotions amp Events ManagerLocation South AfricaType Permanent Hospitality amp EntertainmentBring events to life. Drive guest engagement.This is an...


View Job
Promotions and Events Manager

Johannesburg: Key Accounts Manager – Sub-Sahara Africa – Automotive Industry posted by Employ Africa Group

KEY ACCOUNTS MANAGER AFRICA - Based in South Africa Based in Sandton and reporting into the Company Brand, playing a critical role of developing and...


View Job
Key Accounts Manager – Sub-Sahara Africa – Automotive Industry

Johannesburg South: Receiving Manager (JB5468) posted by Kontak Recruitment

Receiving Manager JB5468Location Steeledale, Johannesburg South, GautengSalary R15 000 - R20 000 per monthEmployment Type PermanentWorkplace On-siteA...


View Job
Receiving Manager (JB5468)

Johannesburg: Electro Mechanical Technician posted by Hire Resolve

Hire Resolves client is currently seeking an experienced Electro-Mechanical Technician to join their dynamic and innovative team in Johannesburg. If youre...


View Job
Electro Mechanical Technician

Johannesburg: Maintenance Manager posted by Hire Resolve

A leadingnbspSouth African manufacturer specializing in plastic profile extrusion and injection molding is looking for anbspMaintenance Manager with...

View Job  Durbanville: Mechanical Revit & CAD Draughtsperson posted by Hire Resolve

View Job
Maintenance Manager

Johannesburg: Logistics Marketing Consultant – Johannesburg posted by Qetello

Join a dynamic team in the logistics industry as a Marketing Consultant, based in vibrant Johannesburg. This exciting role is perfect for a results-driven...


View Job
Logistics Marketing Consultant – Johannesburg

Johannesburg: Semi-skilled Boilermaker posted by Route Management

Work in manufacturing of trucks and trailer environmentLiaising with Production teamsParticipation in the fabrication of metal components including welding,...


View Job
Semi-skilled Boilermaker

Johannesburg: Underwriting Administrator (Roodepoort, JHB) posted by AtripleA recruitment & temps

Job Title Underwriting Administrator Roodepoort, JHB Main Purpose We are seeking a detail-oriented and organized Underwriting Admin to support the...


View Job
Underwriting Administrator (Roodepoort, JHB)

Johannesburg: Sales Representative-In House ( Building Supply) posted by Lumina Personnel

Sales Representative In-House Building Materials.? Alrode, Alberton? Salary Market-Related NegotiableAre you a results-driven sales professional with a...


View Job
Sales Representative-In House ( Building Supply)

Johannesburg: Fleet Manager posted by Cre8work!

Job SummaryAs a Logistics Manager, you will be responsible for overseeing all aspects of logistics operations to ensure the efficient and timely movement of...


View Job
Fleet Manager

Centurion: Creative Manager – Johannesburg posted by PRR Recruitment Services

Creative Manager Digital amp Product Design - Johannesburg Rneg Love bold ideas, sleek interfaces, and the thrill of creating unforgettable digital...


View Job
Creative Manager – Johannesburg

Johannesburg: LAYOUT ARTIST IN BRANDING – FASHION/CORPORATE – FASHION/CORPORATE CLOTHING posted by Top Notch HR Solutions

LAYOUT ARTIST IN BRANDING FASHIONCORPORATE CLOTHINGLocation Kew, JohannesburgGross Salary R14 000, 00 R18000, 00Starting date ASAPType PermanentOur client...


View Job
LAYOUT ARTIST IN BRANDING – FASHION/CORPORATE – FASHION/CORPORATE CLOTHING

Johannesburg: Design Engineer (Roads) posted by Hire Resolve

Are you an experienced Roads Design Engineer looking to make your next career-defining move? Hire Resolve is partnering with a leading multidisciplinary...


View Job
Design Engineer (Roads)

Johannesburg: Mid-Level Developer posted by Hire Resolve

A South African company that provides business intelligence and public relations services, with a strong focus on media monitoring, is looking for a...


View Job
Mid-Level Developer

Johannesburg: PostgreSQL Database Administrator posted by Hire Resolve

Are you a meticulous and experienced PostgreSQL expert with a passion for performance, security, and data integrity? A forward-thinking technology firm is...


View Job
PostgreSQL Database Administrator

Share this to someone who needs a job:
Posted in Jobs in Central Johannesburg, Jobs in Gauteng, Jobs in Johannesburg, Jobs in South Africa
Tagged , ,

More Jobs in Your Area