Menu Close

Durban: Senior Data Engineer posted by Carlysle Human Capital

Senior Data Engineer

Posted on 2025-03-18

Employer Carlysle Human Capital
Salary 0
Category It Computer
Location Kwazulu Natal  /  Durban

Job Summary

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills Design, develop, and maintain the end-to-end technical aspects (hardware – on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers. Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks Optimize data performance, scalability, and security Provide technical guidance and support to data analysts and research scientists. Design data integrations and data quality frameworks Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture Document and effectively communicate data engineering processes and solutions. Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments. Minimum Qualifications: Bachelor`s degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field Industry recognized IT related certification and technology qualification such as Databases and Data related certifications. This is a technical role so a strong focus needs to be on technical skills and experience Minimum Experience: 7+ years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies. Knowledge and Abilities: Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc. Experience of Big Data technologies such as Hadoop, Spark and Hive Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow Experience working with containerization is advantageous Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure. Strong knowledge of data modeling, architecture, and governance principles Strong Linux Administration skills Programming skills in various languages advantageous Strong data security and compliancy experience Excellent communication, collaboration, and problem-solving skills Ability to work independently and as part of a cross-functional team Interest and enthusiasm for medical scientific research and its applications SUMMARY: There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc. You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.
Senior Data Engineer position available in Kwazulu-Natal, Durban. This job position was posted by Carlysle Human Capital. The job has been posted as a premium ad on 2025-03-18 at 13:42:04 in the It Computer category

Click Go Apply to apply online!


You might also like these jobs in the same area.

Apply directly for this position. Please read all instructions carefully.

We do not process job applications; we simply aggregate and display job listings.

More related positions


Johannesburg: Senior Azure Data Engineer – Johannesburg – R500 To R600 Per Hour

An autonomous Namibian bank, providing superior financial services to individual and corporate clients as well as international banking services by liaising directly with financial centres and institutions worldwide is looking for an Azure Data Engineer. T


View Job
Senior Azure Data Engineer – Johannesburg – R500 To R600 Per Hour

Cape Town City Centre: Senior Data Analyst

Key Responsibilities: Design and Development: Develop and maintain advanced analytics applications and data systems. Own key components of data solutions, ensuring they meet the needs of the business. Data Systems Implementation: Lead the development of da


View Job
Senior Data Analyst

Johannesburg: Senior Azure Data Scientist – Johannesburg (Hybrid) – Up To R700 Per Hour

Are you a passionate Senior Data Scientist ready to tackle cutting-edge AI and Data Science challenges? We``re seeking an expert to design and develop Machine Learning (ML) and AI solutions using Microsoft Azure. You``ll work with large datasets, build pre


View Job
Senior Azure Data Scientist – Johannesburg (Hybrid) – Up To R700 Per Hour

Johannesburg: Senior Azure Data Scientist – Johannesburg – 1Mil To 1.1Mil Per Annum

A leading Information Technology and Business Consulting firm, providing specialized solutions to enterprises across the private and public sectors, are looking for a Senior Data Scientist to join their AI & Data Analytics Department. This role offers an e


View Job
Senior Azure Data Scientist – Johannesburg – 1Mil To 1.1Mil Per Annum

Cape Town City Centre: Senior Cloud Data Engineer (Ch1103)

Our client, a leader in the media industry, is looking for Senior Cloud Data Engineer to join their dynamic and fast-paced team, where technology, content, and innovation shape South Africa’s top digital, print, and ecommerce platforms. The client’s team o


View Job
Senior Cloud Data Engineer (Ch1103)

Johannesburg: Senior Data Scientist

Job Opportunity: Senior Data Scientist Are you an experienced and inquisitive individual with a passion for data? Join our team as a Senior Data Scientist and be a key player in shaping our data-driven future Why Join Us? Innovative Environment : Work with


View Job
Senior Data Scientist

Cape Town City Centre: Senior Data Scientist: Client Insights – Contract – Onsite

Our Client a Global tech firm is seeking a Manager: Data and Analytics (Analytics Engineering) to join their team in a banking project on a contract basis in Cape town (Onsite). They offer stability, growth, attract rates and a great working environment Ab


View Job
Senior Data Scientist

Johannesburg: Senior Data Engineer – Johannesburg – R1mil Up To R1,3Mil Per Annum

A global, cutting-edge consultancy, AI solutions developer, with domain expertise in the utilities, insurance, banking, and telecommunications industries and the attitude and aptitude to change the world seek a Senior Data Engineer to join them in their en


View Job
Senior Data Engineer – Johannesburg – R1mil Up To R1,3Mil Per Annum

Johannesburg: Senior Data Analyst

Minimum requirements: Applicable certifications would be advantage Experience working with the following areas in Call Centre environment would be a benefit Credit Card, Personal Loans, Legal up to Judgement granted More then 10 years of experience would b


View Job
Senior Data Analyst

Rivonia: Senior Data Engineer – Contract – Hybrid

Our Client a Global tech firm is seeking 2 Senior Data Engineers to join their in Rivonia on a contract basis (Hybrid). They offer stability, growth, attractive rates and a great working environment. The Data Engineer will be responsible for developing, ma


View Job
Senior Data Engineer – Contract – Hybrid

Rivonia: Senior Data Engineer – Contract – Hybrid

Our Client a Global tech firm is seeking 2 Senior Data Engineers to join their team in Rivonia on a contract basis (Hybrid). They offer stability, growth, attractive rates and a great working environment. The Data Engineer will be responsible for developin


View Job
Senior Data Engineer – Contract – Hybrid

Cape Town City Centre: Senior Data Engineer

The successful candidate will be responsible for building, optimizing, and maintaining complex data pipelines and databases. They will collaborate with data scientists and analysts to ensure seamless data accessibility and quality while developing ETL proc


View Job
Senior Data Engineer

Cape Town City Centre: Senior Data Analyst

The successful candidate will be responsible for gathering, processing, and analyzing data from multiple sources to identify trends and inform business strategies. They will create dashboards, reports, and data visualizations, collaborating with cross-func


View Job
Senior Data Analyst

Cape Town City Centre: Senior Data Scientist

The successful candidate will be responsible for data exploration, feature engineering, and predictive modeling to generate actionable insights. They will lead machine learning and AI initiatives, conduct statistical analyses, and optimize data-driven deci


View Job
Senior Data Scientist

Kuils River: Data Scientist (Junior And Senior Roles) (Ch1109)

Join our client in the credit Industry as a Data Scientist Are you passionate about turning data into actionable insights? Our client is seeking talented Data Scientists on all levels (Senior & Junior), to join their team In this role, you’ll leverage your


View Job
Data Scientist (Junior And Senior Roles) (Ch1109)

Johannesburg: Senior Data Steward posted by PBT Group

PBT Group is seeking a dynamic Data Steward to join our team! In this role, you will ensure adherence to data governance and management standards across business units and enterprise levels. Youll work closely with stakeholders, develop data standards and


View Job
Senior Data Steward

South Africa: Data Scientist (Junior And Senior Roles) (Ch1109) posted by Capital H Staffing and Advisory Solutions

Join our client in the credit Industry as a Data Scientist!Are you passionate about turning data into actionable insights?Our client is seeking talented Data Scientists on all levels (Senior & Junior), to join their team! In this role, youll leverage y


View Job
Data Scientist (Junior And Senior Roles) (Ch1109)

Johannesburg: Senior Data Scientist posted by PBT Group

Job Opportunity: Senior Data ScientistAre you an experienced and inquisitive individual with a passion for data? Join our team as a Senior Data Scientist and be a key player in shaping our data-driven future!Why Join Us?Innovative Environment: Work with cu


View Job
Senior Data Scientist

Cape Town: Senior Cloud Data Engineer (Ch1103) posted by Capital H Staffing and Advisory Solutions

Our client, a leader in the media industry, is looking for Senior Cloud Data Engineer to join their dynamic and fast-paced team, where technology, content, and innovation shape South Africas top digital, print, and ecommerce platforms.The clients team of e


View Job
Senior Cloud Data Engineer (Ch1103)

Cape Town: Senior Data Engineer : Ingestions Squad (Group Data & Ml Platform Tribe) posted by Progressive IT Resourcing

Our ClientCompany Group Technology is responsible for the provision of a digitally enabled technology service as a group COE, drive business and transformation and provide group-wide digital and data architecture. We operate the various technology platform


View Job
Senior Data Engineer

Midrand: Senior Manager Data Engineer posted by Deka Minas (Pty) Ltd

RESPONSIBILITIESCreate and execute a long-term strategy for leveraging data and analytics as a key business enabler and competitive differentiator across the business.Take responsibility for developing and delivering a key element of the organisation`s dat


View Job
Senior Manager Data Engineer

Johannesburg: Senior Manager: Data Science And Business Intelligence posted by Telebest

Our client has an opportunity available for a Senior Manager in the Data Science and Business Intelligence department based in Sandton.Requirements:Postgraduate degree in and Analytical or Qualitative Field.An MBA or additional financial qualification woul


View Job
Senior Manager

South Africa: Senior Microsoft Data Engineer posted by Carlysle Human Capital

5+ years of experience in building Data Warehouse solutions Required skills: Data Warehouse Theory, SQL, ETL/ELT, Project Management Skills required: Snowflake, and Kafka, Azure and MS Fabric Advantageous Skills: Taland, SSIS/SSAS, MS Azure, SAP BW, Qlik,


View Job
Senior Microsoft Data Engineer

Cape Town: Senior Data Analyst posted by Goldman Tech

Job Description:We are seeking a highly skilled and experienced Senior Data Analyst to join one of our Financial Services clients. The ideal candidate will be responsible for analyzing complex data sets and providing valuable insights to inform our busines


View Job
Senior Data Analyst

Cape Town: Senior Data Analyst posted by Goldman Tech

Job Description:We have an amazing opportunity for one of our clients in Bellville - Cape Town.We require a candidate with:• Diploma in Data analysis / Information sciences / IT• At least Matric• 2 or more years in data processing / cleaning


View Job
Senior Data Analyst

Error making API request.
Share this to someone who needs a job:
Posted in Uncategorized

More Jobs in Your Area