Menu Close

Durban: Senior Data Engineer posted by Carlysle Human Capital

Durban: Senior Data Engineer posted by Carlysle Human Capital

Posted on 2025-04-30 00:00:00

Job Summary

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills Design, develop, and maintain the end-to-end technical aspects (hardware – on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers. Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks Optimize data performance, scalability, and security Provide technical guidance and support to data analysts and research scientists. Design data integrations and data quality frameworks Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture Document and effectively communicate data engineering processes and solutions. Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments. Minimum Qualifications: Bachelor’s degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field Industry recognized IT related certification and technology qualification such as Databases and Data related certifications. This is a technical role so a strong focus needs to be on technical skills and experience Minimum Experience: 7+ years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies. Knowledge and Abilities: Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc. Experience of Big Data technologies such as Hadoop, Spark and Hive Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow Experience working with containerization is advantageous Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure. Strong knowledge of data modeling, architecture, and governance principles Strong Linux Administration skills Programming skills in various languages advantageous Strong data security and compliancy experience Excellent communication, collaboration, and problem-solving skills Ability to work independently and as part of a cross-functional team Interest and enthusiasm for medical scientific research and its applications SUMMARY: There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc. You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.

View Job  Paarl: HR Business Partner posted by FROGG Recruitment

Click Go Apply to apply online!


Get Similar Job Alerts
Enter your WhatsApp number (e.g., +27740908132)
You'll receive alerts for new jobs in

Apply directly for this position. Please read all instructions carefully.

We do not process job applications; we simply aggregate and display job listings.

More related positions


Durban: Digital Marketing Manager posted by LMS

Job DescriptionWe are seeking a candidate to join our team and play a pivotal role in designing and standardising our product packaging, creating marketing...


View Job
Digital Marketing Manager

Durban: SALES EXECUTIVE posted by DSS Destiny

Oversee the sales and technical teams in a large manufacturing company. Oversee the sales and technical teams in a large manufacturing company.


View Job
SALES EXECUTIVE

Durban: AUTO ELECTRICIAN posted by DSS Destiny

Diagnostic and RepairDiagnosing and troubleshooting electrical issues in TrucksTrailers.Repairing or replacing faulty electrical components.Assist with...


View Job
AUTO ELECTRICIAN

Durban: Real Estate Agent posted by unique realty

Join Our Expanding Team in Durban!We are looking for vibrant, honest, and hardworking individuals with experience in the real estate industry to join our...


View Job
Real Estate Agent

Durban: Account Manager (JB5103) posted by Kontak Recruitment

Account Manager JB5103jobsinDurban, KZN ? R8kR14k CTCm fuel, phone allowance, laptopJoin a premium FMCG brand specializing in grocery, dairy, and...

View Job  Johannesburg North: Associate Director: International Tax posted by Network Finance

View Job
Account Manager (JB5103)

Durbanville: Sales Manager – Matric Pass posted by Right Kareer

As a Sales Rep, youll be the voice of our brand, communicating with customers, identifying their needs, and delivering a top-notch experience. Youll be the...


View Job
Sales Manager – Matric Pass

Durban North: Call Center Agent posted by Urban Lifestyle Rewards

A well established company in Durban is looking to employ energetic and goal driven call center agents. Basic salary of R5000 and commission up to R25 000.2...


View Job
Call Center Agent

Durbanville: Service Advisor – Automotive posted by Suzuki

We are looking for a Service Advisor in the Tygervalley area to start ASAP.Previous industry experience would be an advantage.This is a high paced, growing...


View Job
Service Advisor – Automotive

Durban Central: Call Centre agents posted by Vital Consultants

Are you a vibrant sales agent looking for a job, feel free to contact us!!!Free training Uncapped commission R1500 attendance bonus Incentives


View Job
Call Centre agents

Durban North: Remote Telesales Agent (UK Life insurance) posted by EKS Solutionz

UK Life Insurance Lead Generation. Minimum 2 years UK outbound experience. Fully remote position entails is you will need a laptop, headset, fibreLTE,...


View Job
Remote Telesales Agent (UK Life insurance)

Durban North: Au Pair posted by Campbell Attorneys

Au pair wanted to assist with 11-year-old sonWe are searching for an au pair to join our family and provide support for our 11-year-old son.Must have the...


View Job
Au Pair

Durban Central: MTN Field Sales Trainer – Immediately needed posted by StokEX Capital

Were seeking motivated and results-driven Sales Trainers to join our dynamic team in Durban. As a Sales Trainer, youll be responsible for driving sales...


View Job
MTN Field Sales Trainer – Immediately needed

Durban North: Region Coordinator – Urgently Needed posted by Webshop

A small company currently located in Briardene is looking for a dynamic and enthusiastic person. Their duties include, doing the day to day admin, filing,...

View Job  Madikwe: Full Lead Trails Guide posted by Kendrick Recruitment

View Job
Region Coordinator – Urgently Needed

Durban: Senior Building Infrastructure Engineer posted by Construct Executive Search

Applicants must have the following requirementsMust have a Bachelor of Science Degree or a Bachelor of Technology Degree or a Bachelor of Engineering Degree...


View Job
Senior Building Infrastructure Engineer

Durban: Senior Water Project Engineer posted by Construct Executive Search

Applicants must have the following requirementsMust have a Bachelor of Science Degree or a Bachelor of Engineering Degree in Civil Engineering.Must be...


View Job
Senior Water Project Engineer

Durban: Electrical Building Services Engineer posted by Construct Executive Search

Applicants must have the following requirementsMust have a Bachelor of Science Degree or a Bachelor of Technology Degree or a Bachelor of Engineering Degree...


View Job
Electrical Building Services Engineer

Durban: Electrical Engineer (Building & Rural) posted by Construct Executive Search

Applicants must have the following requirementsMust have a Bachelors Degree in Electrical Engineering.Must have 5 - 8 years of experience in Electrical...


View Job
Electrical Engineer (Building & Rural)

Durban: Supply Chain Co-Ordinator posted by Bluff Personnel

Supply Chain Co-OrdinatorRef CXP1302025Basic Salary R22 000Temp Position until March 2026 maternity reliefLocation Durban SouthMain Duties include Sales in...


View Job
Supply Chain Co-Ordinator

Durban: Fleet Sales Executive posted by Intercon Recruitment

Dealership in Durban is looking for an experienced Fleet Sales Executive, with previous Motor Dealership Fleet Sales Experience. Salary Basic R9 500 - R13...


View Job
Fleet Sales Executive

Durban: New Vehicle Sales Executive posted by Intercon Recruitment

New Vehicle Sales Executive - Durban - KZNDealership in Durban is looking for an preferably experienced Female, New Vehicle Sales Executive, with previous...


View Job
New Vehicle Sales Executive

Durban: Outbound Telesales Agent (JB5129) posted by Kontak Recruitment

Outbound Telesales Agent JB5129Durban, KwaZulu Natal R8 000 basic plus commission and incentives Drive sales growth by engaging potential clients through...


View Job
Outbound Telesales Agent (JB5129)

Durban: Telesales Manager (JB5131) posted by Kontak Recruitment

Telesales Manager JB5131Durban, KwaZulu Natal R20 000 per month plus performance-based bonuses Category Sales Lead and manage a high-performing telesales...


View Job
Telesales Manager (JB5131)

Durban South: DEBTORS CLERK – URGENTLY HIRING posted by TRULINE TRUCKING SERVICES

Ensuring that client information details are kept up to date.Generating Sales orders.Verifying and confirming load confirmations and rates.Creating and...


View Job
DEBTORS CLERK – URGENTLY HIRING

Durban South: Admin Analyst Assistant – Urgently needed posted by South African Inland Logistics

We are seeking a highly skilled and detail-oriented Admin Analyst Assistant to join our team. The ideal candidate must be exceptionally proficient in...


View Job
Admin Analyst Assistant – Urgently needed

Durban North: Call Centre Agents posted by Urban Lifestyle Rewards

Urban Lifestyle Rewards based in Durban is looking to employ energetic and goal driven call center agents. Basic salary of R5000 and commission up to R25...


View Job
Call Centre Agents

Share this to someone who needs a job:
Posted in Jobs in Durban, Jobs in Kwazulu-Natal, Jobs in South Africa

More Jobs in Your Area