Menu Close

Durban: Senior Data Engineer

Date 2025-01-18
Job Title Senior Data Engineer
Employer
More Information
Salary Monthly
Category It Computing Software
Location eThekwini
/ Durban

Job Summary

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills Design, develop, and maintain the end-to-end technical aspects (hardware – on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers. Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks Optimize data performance, scalability, and security Provide technical guidance and support to data analysts and research scientists. Design data integrations and data quality frameworks Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture Document and effectively communicate data engineering processes and solutions. Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments. Minimum Qualifications: Bachelor`s degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field Industry recognized IT related certification and technology qualification such as Databases and Data related certifications. This is a technical role so a strong focus needs to be on technical skills and experience Minimum Experience: 7 years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies. Knowledge and Abilities: Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc. Experience of Big Data technologies such as Hadoop, Spark and Hive Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow Experience working with containerization is advantageous Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure. Strong knowledge of data modeling, architecture, and governance principles Strong Linux Administration skills Programming skills in various languages advantageous Strong data security and compliancy experience Excellent communication, collaboration, and problem-solving skills Ability to work independently and as part of a cross-functional team Interest and enthusiasm for medical scientific research and its applications SUMMARY: There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc. You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.

View Job  Westville: Corporate Services Administrator

Senior Data Engineer position available in eThekwini, Durban. This job position was posted by . The job has been posted on 2025-01-18 in the It Computing Software category

Click Go Apply to apply online!


You might also like to look at the available jobs in the same area.

Apply directly for this position
Some posts have contact information included with other instructions. All applicants are urged to read the instructions carefully.
We will list jobs at the bottom of each job, so feel free to scroll down and continue your search.

We do not accept any job applications and merely offer the service of daily jobs being mined and displayed on our job portal.
Each job post has a link to take you directly to the original post to apply on their website or the original website where the post came from.


Here are some more related positions


Johannesburg: Senior Data Analyst – Experienced posted by All Star Placements

We have a great position available for a Senior Data Analyst in the Woodmead area for a head office in the mining industry.Candidates will be expected to: Monthly Reports - Preparation of all reports required for mine month end processing. Monthly Manageme


View Job
Senior Data Analyst – Experienced

Cape Town City Centre: Senior Data Analyst

Key Responsibilities: Design and Development: Develop and maintain advanced analytics applications and data systems. Own key components of data solutions, ensuring they meet the needs of the business. Data Systems Implementation: Lead the development of da


View Job
Senior Data Analyst

Cape Town City Centre: Senior Software Engineer – Data, Group Data & Ml Platform Tribe

What will you do ? Our client is a newly created engineering organisation on a mission to empower and accelerate autonomous value creation through insights and ML across the group. The company main users are the data scientists and analysts who sit inside


View Job
Senior Software Engineer – Data, Group Data & Ml Platform Tribe

Cape Town City Centre: Senior Hubspot Crm & Data Specialist

Who are we? A fast-paced, innovation-driven company specialising in spend management. They develop cloud-based software that simplifies procurement processes for mid-sized companies, enabling them to manage, optimise, and analyse business spend effectively


View Job
Senior Hubspot Crm & Data Specialist

Cape Town City Centre: Senior Project Manager – Investment Data Projects

Job Description This progressive insurance company is on the lookout for an individual to join their projects department as a Senior Project Manager - Investment Data Projects. The suitable candidate will need to have a keen interest in working within the

View Job  Durban: Administrator / Coordinator (Finance) posted by AtripleA recruitment & temps

View Job
Senior Project Manager – Investment Data Projects

Menlyn: Aiqx Data Scientist Specialist (Senior) 1785

Essential Skills Requirements: Do you have what it takes? Were looking for exceptional talent with: Strong expertise in Applied Machine Learning , Data Science , and Deep Learning . Proven experience building predictive models using frameworks like TensorF


View Job
Aiqx Data Scientist Specialist (Senior) 1785

Johannesburg: Senior Teradata Data Engineer – Contract – Sandton – R680 Per Hour

We are currently in search for a Data Engineer with 5 to 8 years experience in data engineering and strong backend and front-end development skills. You will be required to build data warehouse components on Teradata warehouse (3NF) and will have to comply


View Job
Senior Teradata Data Engineer – Contract – Sandton – R680 Per Hour

Cape Town: Senior Hubspot Crm & Data Specialist posted by Ad Talent Africa

Who are we?A fast-paced, innovation-driven company specialising in spend management. They develop cloud-based software that simplifies procurement processes for mid-sized companies, enabling them to manage, optimise, and analyse business spend effectively.


View Job
Senior Hubspot Crm & Data Specialist

Cape Town: Senior Data Engineer : Ingestions Squad (Group Data & Ml Platform Tribe) posted by Progressive IT Resourcing

Our ClientCompany Group Technology is responsible for the provision of a digitally enabled technology service as a group COE, drive business and transformation and provide group-wide digital and data architecture. We operate the various technology platform


View Job
Senior Data Engineer

Pretoria: Senior Manager: Digital And Data posted by People Dimension

Job Description/Duties: Formulation of a digital strategy Manage third-party IT providers and the development of data architecture. Manage and coordinate the development of bespoke systems, digital channels and other software development. Ensure solutions


View Job
Senior Manager

Midrand: Senior Manager Data Engineer posted by Deka Minas (Pty) Ltd

RESPONSIBILITIESCreate and execute a long-term strategy for leveraging data and analytics as a key business enabler and competitive differentiator across the business.Take responsibility for developing and delivering a key element of the organisation`s dat


View Job
Senior Manager Data Engineer

Johannesburg: Senior Manager: Data Science And Business Intelligence posted by Telebest

Our client has an opportunity available for a Senior Manager in the Data Science and Business Intelligence department based in Sandton.Requirements:Postgraduate degree in and Analytical or Qualitative Field.An MBA or additional financial qualification woul


View Job
Senior Manager

Johannesburg: Senior Data Analytics Specialist posted by IOCO

We are seeking a Senior Data Analytics Specialist with excellent problem-solving skills, attention to detail, and the ability to translate data insights into strategic business recommendations. You will be responsible for providing data-driven insights an


View Job
Senior Data Analytics Specialist

Durban: Senior Data Engineer posted by Carlysle Human Capital

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutio


View Job
Senior Data Engineer

Cape Town: Senior Data Analyst posted by Goldman Tech

Job Description:We have an amazing opportunity for one of our clients in Bellville - Cape Town.We require a candidate with:• Diploma in Data analysis / Information sciences / IT• At least Matric• 2 or more years in data processing / cleaning


View Job
Senior Data Analyst

Cape Town: Senior Data Analyst posted by Goldman Tech

Job Description:We are seeking a highly skilled and experienced Senior Data Analyst to join one of our Financial Services clients. The ideal candidate will be responsible for analyzing complex data sets and providing valuable insights to inform our busines

View Job  Durban: Procurement Manager posted by Goldman Tech

View Job
Senior Data Analyst

South Africa: Senior Data Engineer posted by Goldman Tech

Job Description:We have an amazing opportunity for a Senior Data EngineerWe require a candidate with:• Degree in Data Science, Engineering, Computer Science or Math.• Cloud certifications advantageous.• 5-10 years of experience in a data eng


View Job
Senior Data Engineer

Johannesburg: Senior Data Analyst posted by Goldman Tech

Job Description:One of our clients in Johannesburg is searching for an innovative senior data analyst to join their team. We require a candidate with:• Post matric qualifications (Quantitative or IT subjects are advantageous• 3 + years working ex


View Job
Senior Data Analyst

Johannesburg: Senior Data Warehouse Developer posted by Goldman Tech

Job Description:We have an amazing opportunity for a Senior Data Warehouse Developer in Johannesburg!We require a candidate with:• BSc. Computer Sciences or National Diploma in IT• 5+ years experience in a data warehousing role• 4+ years wri


View Job
Senior Data Warehouse Developer

Cape Town: Senior Sql Database Administrator posted by Goldman Tech

Job Description:We have an amazing opportunity for a Senior SQL Database Administrator in Cape Town!We require a candidate with:• A relevant 3-year Information Technology Degree or Diploma• Minimum of 10 years` experience in a similar role •


View Job
Senior Sql Database Administrator

Cape Town: Senior Data Analyst posted by Goldman Tech

Job Description:We have an amazing opportunity for a Senior Data Analyst in Cape Town! We require a candidate with experience in:• MS Excel• SQL query writing• Big data reporting packages (Power BI, Qlikview, Tableau)Should you meet the requ


View Job
Senior Data Analyst

Johannesburg: Senior Data Warehouse/ Bi Developer posted by Goldman Tech

Job Description:We have an amazing opportunity for a Senior Data Warehouse/ BI Developer in Johannesburg!We require a candidate with experience in:• BI Toolsets• ETL Applications• T-SQL• Implementation of DW/BI environments• BI Too


View Job
Senior Data Warehouse/ Bi Developer

Cape Town: Senior Data Scientist posted by Goldman Tech

Job Description:We have an exciting client in Cape Town looking for a Senior Data Scientist.You will:Build segmentation and propensity modelsBuild real time systems that integrate with the rest of our stackBuild visualisations using R Shiny, DataStudio, Ta


View Job
Senior Data Scientist

Cape Town: Senior Data Scientist posted by Goldman Tech

Job Description:We have a client in Cape Town looking for a Senior Data Scientist.You will:Build state-of-the-art statistical modelsBuild segmentation and propensity modelsBuild real time systems that integrate with the rest of our stackBuild visualisation


View Job
Senior Data Scientist

Cape Town: Senior Database Administrator posted by Goldman Tech

Job Description:Seeking a Senior Database Administrator in Cape Town for a Well-Known IT Company!Requirements:• National Diploma in Information Technology or Equivalent• 8+ Years` experience as a Database Administrator • Recent experience wi


View Job
Senior Database Administrator

Error making API request.
Share this to someone who needs a job:
Posted in Jobs in Durban, Jobs in eThekwini

More Jobs in Your Area