Johannesburg: Senior Data Engineer – JHB posted by Hire Resolve
Posted on 2025-08-29 00:00:00
Employer | Hire Resolve |
---|---|
Category | Other IT/Computer |
Location | Gauteng / Johannesburg |
Job Summary
This
role requires expertise in data modeling, ETL development, stream processing, and distributed data systems. The ideal candidate will work closely with developers, DevOps, and analytics teams to transform raw network data into clean, structured, and query–ready datasets that power dashboards, machine learning models, and business logic. The candidate will provide technical leadership, optimize data workflows for performance and reliability, and drive best practices in data engineering methodologiesResponsibilities:
- Build ETL/ELT pipelines for ingesting, cleansing, and transforming CDRs and telecommunications logs
from multiple network elements (5G/4G/3G/2G). - Design and maintain real-time data flows using Kafka, Apache NiFi and Apache Flink.
- Work with large-scale distributed file systems for batch and streaming ingestion.
- Integrate and structure data for analytics platform like Apache Druid, Hudi and Superset.
- Develop CI/CD pipelines for deploying data workflows and transformation logic.
- Ensure data quality, schema validation and compliance with retention and security policies.
- Monitor data pipeline health and optimize performance throughput and cost efficiency.
- Write complex and performant queries for data validation, transformation, aggregation, and analytics
across relational and distributed platforms. - Develop and optimize big data processing workflows in platforms like Apache Spark, Hive and Druid.
- Establish efficient issue tracking and workflow processes, enhancing productivity and collaboration
across engineering teams. - Implement security best practices and compliance frameworks to safeguard infrastructure, data, and
applications from vulnerabilities and threats. - Maintain secure role-based access control mechanisms, encryption strategies, and identity management
solutions to protect sensitive data and ensure regulatory compliance. - Mapping data flows from source to transformation to consumption.
- Design and implement full-text search and indexing solutions for querying and retrieval of structured and unstructured telecommunications data using Apache tools or similar search engines.
- Analyze and estimate and implement storage requirements and strategies for large-scale CDR datasets
and real-time data streams, ensuring optimal resource allocation and scalability across environments. - Ensure data integrity and consistency across ingestion, transformation and storage layers through
validation checks, schema enforcement and robust error handling mechanisms. - Develop and maintain quality monitoring tools to proactively detect anomalies, missing records or data
corruption across pipelines. - Perform other duties as assigned
Requirements:
- Bachelors degree in Computer Engineering, Software Engineering, Computer Science, or a related field.
- Strong experience in building data pipelines using tools like Apache NiFi, Kafka, Airflow or similar.
- Proficiency in SQL, Python, database administration and management like PostgreSQL, MySQL.
-
Solid understanding of distributed data systems like Hive and Hudi and Spark.
-
Experience with streaming frameworks like Kafka Streams, Apache Flink and Apache Beam.
-
Familiarity with data serialization formats like JSON.
- Knowledge of SFTP and secure data transfer mechanisms for ingesting remote files.
- Proficient with Linux environments, shell scripting and storage systems like Ceph.
- Experience with data governance, including data privacy and regulatory compliance like GDPR and
implementing access control, auditing and data usage policies. - Experience in maintaining central inventory of data assets, managing metadata and enabling searchable
discovery across structured and unstructured datasets. - Experience in data lineage tracking to map data flows, visualize and track dependencies.
- Experience with OLAP systems, analytical modelling and columnar databases and designing and
querying multidimensional cubes. - Strong problem–solving skills, ability to work in a fast–paced environment, and manage multiple projects
efficiently. - Strong collaboration skills, adaptability, and a commitment to continuous learning
How to Apply:
If you would like to apply for this role, kindly forward your CV to Gaby Turner at *****@*****.co.za or you may forward your CV to *****@*****.co.za
Click Go Apply to apply online!
Get Similar Job Alerts
More jobs in Johannesburg
More jobs in Gauteng
Browse Other IT/Computer Jobs
Browse Hire Resolve Jobs
Apply directly for this position. Please read all instructions carefully.
We do not process job applications; we simply aggregate and display job listings.
More related positions
Johannesburg: Content Creator posted by Liham Consulting Pty (Ltd)
Job DescriptionPreparing print-ready designs, including formatting, resizing, and image adjustments while ensuring brand and colour consistency.Creating...
View Job
Content Creator
Johannesburg: Supply Chain Coordinator posted by Headhunters
Our client, a leader in the automotive component manufacturing sector, is currently looking to employ a Supply Chain Coordinator based in Elandsfontein,...
View Job
Supply Chain Coordinator
Johannesburg: Project Manager posted by Daisy JHB (Pty) Ltd
Purpose of the roleThe Project Manager is responsible for planning, organizing, and executing projects, ensuring they are completed on time, within budget,...
View Job
Project Manager
Johannesburg: Customer Service Representative – Mbombela posted by Tailormade Solution
Drive sales efforts through effective planning, scheduling and conducting ongoing customer visitsEngage with existing customer base in order to grow...
View Job
Customer Service Representative – Mbombela
Johannesburg: Part Sales Executive – Wanted immediately posted by First degree Recruitment
We are seeking a dynamic and customer-focused Part Sales Executive to join our team. In this role, you will be responsible for selling automotive parts to...
View Job
Part Sales Executive – Wanted immediately
Johannesburg: Exports Logistics Coordinator – Johannesburg posted by Hire Resolve
A leading manufacturer and distributor in the FMCG sector is seeking an experienced Exports Logistics Coordinator to join their fast-paced team based in...
View Job
Exports Logistics Coordinator – Johannesburg
Johannesburg South: PROPERTY ADMINISTRATOR – REQUIRED URGENTLY posted by ERZ Telecoms
- Debtors Management Administration- Creditors Management Administration- Assisting with Drawing up of leases- Rent Roll-MDA experience will be...
View Job
PROPERTY ADMINISTRATOR – REQUIRED URGENTLY
Johannesburg: Oracle Finance Functional Consultant posted by Hire Resolve
? Oracle Finance Functional Consultantnbsp ? LocationnbspJohannesburg ? Rate Competitive Based on experience? Start Immediate NegotiableA leading...
View Job
Oracle Finance Functional Consultant
Johannesburg: Maintenance Manager – Johannesburg posted by Hire Resolve
A leading manufacturer in the plastics and packaging industry is seeking an experienced Maintenance Manager with strong mechanical and electrical expertise...
View Job
Maintenance Manager – Johannesburg
Johannesburg: Mid-Level Software Developers posted by Hire Resolve
An established PR and media company based in Johannesburg is seeking the expertise of two Mid-Level Software Developers to join their growing remote...
View Job
Mid-Level Software Developers
Johannesburg: Promotions and Events Manager posted by Edgexec (Pty) Ltd
Promotions amp Events ManagerLocation South AfricaType Permanent Hospitality amp EntertainmentBring events to life. Drive guest engagement.This is an...
View Job
Promotions and Events Manager
Johannesburg: Key Accounts Manager – Sub-Sahara Africa – Automotive Industry posted by Employ Africa Group
KEY ACCOUNTS MANAGER AFRICA - Based in South Africa Based in Sandton and reporting into the Company Brand, playing a critical role of developing and...
View Job
Key Accounts Manager – Sub-Sahara Africa – Automotive Industry
Johannesburg South: Receiving Manager (JB5468) posted by Kontak Recruitment
Receiving Manager JB5468Location Steeledale, Johannesburg South, GautengSalary R15 000 - R20 000 per monthEmployment Type PermanentWorkplace On-siteA...
View Job
Receiving Manager (JB5468)
Johannesburg: Electro Mechanical Technician posted by Hire Resolve
Hire Resolves client is currently seeking an experienced Electro-Mechanical Technician to join their dynamic and innovative team in Johannesburg. If youre...
View Job
Electro Mechanical Technician
Johannesburg: Maintenance Manager posted by Hire Resolve
A leadingnbspSouth African manufacturer specializing in plastic profile extrusion and injection molding is looking for anbspMaintenance Manager with...
View Job
Maintenance Manager
Johannesburg: Logistics Marketing Consultant – Johannesburg posted by Qetello
Join a dynamic team in the logistics industry as a Marketing Consultant, based in vibrant Johannesburg. This exciting role is perfect for a results-driven...
View Job
Logistics Marketing Consultant – Johannesburg
Johannesburg: Semi-skilled Boilermaker posted by Route Management
Work in manufacturing of trucks and trailer environmentLiaising with Production teamsParticipation in the fabrication of metal components including welding,...
View Job
Semi-skilled Boilermaker
Johannesburg: Underwriting Administrator (Roodepoort, JHB) posted by AtripleA recruitment & temps
Job Title Underwriting Administrator Roodepoort, JHB Main Purpose We are seeking a detail-oriented and organized Underwriting Admin to support the...
View Job
Underwriting Administrator (Roodepoort, JHB)
Johannesburg: Sales Representative-In House ( Building Supply) posted by Lumina Personnel
Sales Representative In-House Building Materials.? Alrode, Alberton? Salary Market-Related NegotiableAre you a results-driven sales professional with a...
View Job
Sales Representative-In House ( Building Supply)
Johannesburg: Fleet Manager posted by Cre8work!
Job SummaryAs a Logistics Manager, you will be responsible for overseeing all aspects of logistics operations to ensure the efficient and timely movement of...
View Job
Fleet Manager
Centurion: Creative Manager – Johannesburg posted by PRR Recruitment Services
Creative Manager Digital amp Product Design - Johannesburg Rneg Love bold ideas, sleek interfaces, and the thrill of creating unforgettable digital...
View Job
Creative Manager – Johannesburg
Johannesburg: LAYOUT ARTIST IN BRANDING – FASHION/CORPORATE – FASHION/CORPORATE CLOTHING posted by Top Notch HR Solutions
LAYOUT ARTIST IN BRANDING FASHIONCORPORATE CLOTHINGLocation Kew, JohannesburgGross Salary R14 000, 00 R18000, 00Starting date ASAPType PermanentOur client...
View Job
LAYOUT ARTIST IN BRANDING – FASHION/CORPORATE – FASHION/CORPORATE CLOTHING
Johannesburg: Design Engineer (Roads) posted by Hire Resolve
Are you an experienced Roads Design Engineer looking to make your next career-defining move? Hire Resolve is partnering with a leading multidisciplinary...
View Job
Design Engineer (Roads)
Johannesburg: Mid-Level Developer posted by Hire Resolve
A South African company that provides business intelligence and public relations services, with a strong focus on media monitoring, is looking for a...
View Job
Mid-Level Developer
Johannesburg: PostgreSQL Database Administrator posted by Hire Resolve
Are you a meticulous and experienced PostgreSQL expert with a passion for performance, security, and data integrity? A forward-thinking technology firm is...
View Job
PostgreSQL Database Administrator