Menu Close

Durban: Senior Data Engineer posted by Carlysle Human Capital

Senior Data Engineer

Posted on 2025-04-16

Employer Carlysle Human Capital
Salary 0
Category It Computer
Location Kwazulu Natal  /  Durban

Job Summary

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills Design, develop, and maintain the end-to-end technical aspects (hardware – on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers. Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks Optimize data performance, scalability, and security Provide technical guidance and support to data analysts and research scientists. Design data integrations and data quality frameworks Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture Document and effectively communicate data engineering processes and solutions. Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments. Minimum Qualifications: Bachelor`s degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field Industry recognized IT related certification and technology qualification such as Databases and Data related certifications. This is a technical role so a strong focus needs to be on technical skills and experience Minimum Experience: 7+ years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies. Knowledge and Abilities: Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc. Experience of Big Data technologies such as Hadoop, Spark and Hive Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow Experience working with containerization is advantageous Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure. Strong knowledge of data modeling, architecture, and governance principles Strong Linux Administration skills Programming skills in various languages advantageous Strong data security and compliancy experience Excellent communication, collaboration, and problem-solving skills Ability to work independently and as part of a cross-functional team Interest and enthusiasm for medical scientific research and its applications SUMMARY: There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc. You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.
Senior Data Engineer position available in Kwazulu-Natal, Durban. This job position was posted by Carlysle Human Capital. The job has been posted as a premium ad on 2025-04-16 at 13:43:02 in the It Computer category

Click Go Apply to apply online!


You might also like these jobs in the same area.

Apply directly for this position. Please read all instructions carefully.

We do not process job applications; we simply aggregate and display job listings.

More related positions


Johannesburg: Senior Data Engineer – Johannesburg – R1mil Up To R1,3Mil Per Annum

A global, cutting-edge consultancy, AI solutions developer, with domain expertise in the utilities, insurance, banking, and telecommunications industries and the attitude and aptitude to change the world seek a Senior Data Engineer to join them in their en


View Job
Senior Data Engineer – Johannesburg – R1mil Up To R1,3Mil Per Annum

Johannesburg: Senior Data Analyst

Minimum requirements: Applicable certifications would be advantage Experience working with the following areas in Call Centre environment would be a benefit Credit Card, Personal Loans, Legal up to Judgement granted More then 10 years of experience would b


View Job
Senior Data Analyst

Rivonia: Senior Data Engineer – Contract – Hybrid

Our Client a Global tech firm is seeking 2 Senior Data Engineers to join their in Rivonia on a contract basis (Hybrid). They offer stability, growth, attractive rates and a great working environment. The Data Engineer will be responsible for developing, ma


View Job
Senior Data Engineer – Contract – Hybrid

Rivonia: Senior Data Engineer – Contract – Hybrid

Our Client a Global tech firm is seeking 2 Senior Data Engineers to join their team in Rivonia on a contract basis (Hybrid). They offer stability, growth, attractive rates and a great working environment. The Data Engineer will be responsible for developin


View Job
Senior Data Engineer – Contract – Hybrid

Johannesburg: Senior Data Scientist (Ai, Machine Learning) – Johannesburg – R1mil To R1.3Mil Ctc

An African financial services organisation that offers a broad spectrum of financial solutions and offers great culture, leadership, learning opportunities, challenges, rewards and recognition to its employees seeks a Senior Data Scientist who will be acco


View Job
Senior Data Scientist (Ai, Machine Learning) – Johannesburg – R1mil To R1.3Mil Ctc

Cape Town City Centre: Senior Data Scientist

Welcome to the big leagues of data science This isnt just crunching numbersthis is revolutionizing how people shop. As a Senior Data Scientist, youll dive deep into complex datasets, uncover hidden insights, and deploy machine learning models to transform


View Job
Senior Data Scientist

Johannesburg: Senior Azure Data Engineer – Johannesburg – R500 To R600 Per Hour

A leading autonomous Namibian bank, committed to providing superior financial services to individual and corporate clients, as well as international banking services through global financial centres, is seeking a Senior Azure Data Engineer . This role offe


View Job
Senior Azure Data Engineer – Johannesburg – R500 To R600 Per Hour

Cape Town City Centre: Senior Data Analyst

Are you a data enthusiast with a passion for turning complex data into actionable insights? A leading company in the retail tech space is looking for a Senior Data Analyst to join their innovative data team . In this remote role, you will play a pivotal pa


View Job
Senior Data Analyst

South Africa: Senior Data Science Consultant

Our client is seeking a full-time Senior Data Science Consultant with strong data science and consulting expertise to join their dynamic and growing Data Science team. Role and Responsibilities: As part of the company Data Science team, you will report dir


View Job
Senior Data Science Consultant

Midrand: Senior Credit Data Analyst / Credit Data And Risk Insights Analyst

About the Role Fempower is recruiting for a highly skilled Senior Credit Data Analyst on behalf of a leading financial institution. This role is pivotal in sourcing, aggregating, and analyzing credit data to support risk modeling, pricing, and portfolio re


View Job
Senior Credit Data Analyst / Credit Data And Risk Insights Analyst

Johannesburg: Senior Data Engineer

Key Responsibilities: Architect and build highly scalable distributed systems using open-source tools. Design and implement efficient data models, with a deep understanding of various data structures and their benefits and limitations. Work with big data b


View Job
Senior Data Engineer

Cape Town City Centre: Senior Data Engineer

Key Responsibilities: Architect and build highly scalable distributed systems using open-source tools. Design and implement efficient data models, with a deep understanding of various data structures and their benefits and limitations. Work with big data b


View Job
Senior Data Engineer

Cape Town City Centre: Senior Data Engineer

Job Specifics: This is a remote role, open to applicants that reside in Africa or South Africa. Travel may be required from time to time, expenses will be covered by the company. About the Client: Our client, a dynamic and forward-thinking global organizat


View Job
Senior Data Engineer

Johannesburg: Senior Data Engineer – Contract – Sandton – R650 To R680 Per Hour

This role is ideal for a professional who thrives on designing robust data solutions while ensuring governance and creating impactful data visualizations. Does this sound like a role youd like to pursue? I can assist you further This position calls for a s


View Job
Senior Data Engineer – Contract – Sandton – R650 To R680 Per Hour

Johannesburg: Senior Data Analyst (Ai Project Experience) – Johannesburg – R700k To 900K Per Annum

An AI-driven solutions company is seeking a Senior Data Analyst to drive strategic decision-making and shape the future of our organization. This role offers the opportunity to work on diverse projects, collaborate with cross-functional teams, and expand e


View Job
Senior Data Analyst (Ai Project Experience) – Johannesburg – R700k To 900K Per Annum

South Africa: Senior Data Scientist (Cpt)

ENVIRONMENT: JOIN the Cape Town Division of a dynamic Financial Services provider seeking the expertise of a Senior Data Scientist who will be responsible for implementing and managing analytical projects at various public sector or private sector entities


View Job
Senior Data Scientist (Cpt)

Johannesburg: Senior Data Engineer

Senior Data Enbgineer required in Sandton for a contract. This role is pivotal in driving our data strategy and ensuring the efficient and effective management of our data infrastructure. We are looking for a proactive, innovative, and solution-oriented in


View Job
Senior Data Engineer

Johannesburg: Senior Azure Data Scientist – Johannesburg – 1Mil To 1.1Mil Per Annum

A leading Information Technology and Business Consulting firm, providing specialized solutions to enterprises across the private and public sectors, are looking for a Senior Data Scientist to join their AI & Data Analytics Department. This role offers an e


View Job
Senior Azure Data Scientist – Johannesburg – 1Mil To 1.1Mil Per Annum

Johannesburg: Senior Technical Data Analyst posted by PBT Group

A Senior Technical Data Analyst is required to support data-driven initiatives within a dynamic banking environment. The ideal candidate will bring strong analytical expertise, hands-on technical skills, and experience working with cloud platforms (especia


View Job
Senior Technical Data Analyst

Johannesburg: Senior Data Engineer posted by Deka Minas (Pty) Ltd

Senior Data Enbgineer required in Sandton for a contract. This role is pivotal in driving our data strategy and ensuring the efficient and effective management of our data infrastructure. We are looking for a proactive, innovative, and solution-oriented in


View Job
Senior Data Engineer

Cape Town: Senior Data Engineer posted by CodeConnect Staffing (Pty) Ltd

Job Specifics: This is a remote role, open to applicants that reside in Africa or South Africa.Travel may be required from time to time, expenses will be covered by the company.About the Client:Our client, a dynamic and forward-thinking global organization


View Job
Senior Data Engineer

Midrand: Senior Credit Data Analyst / Credit Data And Risk Insights Analyst posted by Fempower Personnel

About the RoleFempower is recruiting for a highly skilled Senior Credit Data Analyst on behalf of a leading financial institution. This role is pivotal in sourcing, aggregating, and analyzing credit data to support risk modeling, pricing, and portfolio rep


View Job
Senior Credit Data Analyst / Credit Data And Risk Insights Analyst

Cape Town: Senior Aws Data Engineer posted by PBT Group

The Data Engineer will be responsible for designing, implementing, and optimising data pipelines and data solutions using AWS and Snowflake. The successful candidate will work closely with Data Analysts, Data Scientists, and other stakeholders to ensure ef


View Job
Senior Aws Data Engineer

South Africa: Data Scientist (Junior And Senior Roles) (Ch1109) posted by Capital H Staffing and Advisory Solutions

Join our client in the credit Industry as a Data Scientist!Are you passionate about turning data into actionable insights?Our client is seeking talented Data Scientists on all levels (Senior & Junior), to join their team! In this role, youll leverage y


View Job
Data Scientist (Junior And Senior Roles) (Ch1109)

Cape Town: Senior Cloud Data Engineer (Ch1103) posted by Capital H Staffing and Advisory Solutions

Our client, a leader in the media industry, is looking for Senior Cloud Data Engineer to join their dynamic and fast-paced team, where technology, content, and innovation shape South Africas top digital, print, and ecommerce platforms.The clients team of e


View Job
Senior Cloud Data Engineer (Ch1103)

Error making API request.
Share this to someone who needs a job:
Posted in Uncategorized

More Jobs in Your Area