Menu Close

Durban: Senior Data Engineer posted by Carlysle Human Capital

Durban: Senior Data Engineer posted by Carlysle Human Capital

Posted on 2025-05-21 00:00:00

Job Summary

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills Design, develop, and maintain the end-to-end technical aspects (hardware – on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers. Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks Optimize data performance, scalability, and security Provide technical guidance and support to data analysts and research scientists. Design data integrations and data quality frameworks Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture Document and effectively communicate data engineering processes and solutions. Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments. Minimum Qualifications: Bachelor’s degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field Industry recognized IT related certification and technology qualification such as Databases and Data related certifications. This is a technical role so a strong focus needs to be on technical skills and experience Minimum Experience: 7+ years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies. Knowledge and Abilities: Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc. Experience of Big Data technologies such as Hadoop, Spark and Hive Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow Experience working with containerization is advantageous Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure. Strong knowledge of data modeling, architecture, and governance principles Strong Linux Administration skills Programming skills in various languages advantageous Strong data security and compliancy experience Excellent communication, collaboration, and problem-solving skills Ability to work independently and as part of a cross-functional team Interest and enthusiasm for medical scientific research and its applications SUMMARY: There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc. You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.

View Job  Johannesburg: Electrical Engineer/Electrician posted by Deka Minas (Pty) Ltd

Click Go Apply to apply online!


Get Similar Job Alerts
Enter your WhatsApp number (e.g., +27740908132)
You'll receive alerts for new jobs in Durban, Kwazulu-Natal

Apply directly for this position. Please read all instructions carefully.

We do not process job applications; we simply aggregate and display job listings.

More related positions


Durban South: DEBTORS CLERK – URGENTLY HIRING posted by TRULINE TRUCKING SERVICES

Ensuring that client information details are kept up to date.Generating Sales orders.Verifying and confirming load confirmations and rates.Creating and...


View Job
DEBTORS CLERK – URGENTLY HIRING

Durban South: Admin Analyst Assistant – Urgently needed posted by South African Inland Logistics

We are seeking a highly skilled and detail-oriented Admin Analyst Assistant to join our team. The ideal candidate must be exceptionally proficient in...


View Job
Admin Analyst Assistant – Urgently needed

Durban North: Call Centre Agents posted by Urban Lifestyle Rewards

Urban Lifestyle Rewards based in Durban is looking to employ energetic and goal driven call center agents. Basic salary of R5000 and commission up to R25...


View Job
Call Centre Agents

Durban North: Technical Sales Representative posted by Brasco KwaZulu-Natal

Brasco Kwazulu Natal is looking for a dynamic and driven Technical Sales Representative to join our KZN branch. The ideal candidate will have strong...

View Job  Cape Town: Senior Data Scientist (CPT) posted by Datafin

View Job
Technical Sales Representative

Durbanville: Real Estate Canvasser posted by Just Property

Were Hiring! Real Estate Canvasser Wanted Just Property Dynamic is looking for a driven and energetic Canvasser to join our team! If you have a passion for...


View Job
Real Estate Canvasser

Durban South: Commercial Manager posted by Bluff Personnel

COMMERCIAL MANAGERRef CXP1322025An exciting opportunity has arisen for an experienced and suitably qualified Commercial Manager Salary Range R72 000...


View Job
Commercial Manager

Randburg: SALES AGENT posted by Real Promotions Durban

Job OverviewWe are looking for an enthusiastic and results-driven Sales Agent to join our dynamic team. As a sales agent, you will be responsible for...


View Job
SALES AGENT

Durban: Accounts Administrator – Needed Urgently posted by Tactical Security Services

QUALIFICATIONSMinimum - 5 years experience in the accounts field. Matric certificateDrivers license own car preferablyAccounting Book-keeping...


View Job
Accounts Administrator – Needed Urgently

Durban: Fleet Sales Executive – Durban – KZN posted by Intercon Recruitment

Dealership in Durban is looking for an experienced Fleet Sales Executive, with previous Motor Dealership Fleet Sales Experience. Salary Basic R9 500 - R13...


View Job
Fleet Sales Executive – Durban – KZN

Durban: New Vehicle Sales Executive posted by Intercon Recruitment

Dealership in Durban is looking for an experienced Female, New Vehicle Sales Executive, with previous Motor Dealership New Vehicle Sales Experience. Salary...


View Job
New Vehicle Sales Executive

Durban: Personal Assistant posted by Aurora Gynaecology

About the RoleWe are seeking a highly organized and dedicated Personal Assistant to provide comprehensive support to a Gynaecological Oncologist. This role...


View Job
Personal Assistant

Durban North: Printing Operator – Urgently needed posted by Contact Print Business Forms

Printing company based in Redhill industrial area looking for an experienced printing machine operator for a continuous forms machine.

View Job  Midrand: Manager: Regulatory Affairs | Midrand posted by The Recruitment Council

View Job
Printing Operator – Urgently needed

Durban: Business Development Consultant posted by NP recruitment

Urgent - 01 May 2025 Intake.Job Vacancy!!! New Business Development Consultants RequiredLocation Durban BasedShort Term Insurance Sector Property and...


View Job
Business Development Consultant

Durbanville: Electrical Assistant posted by GJG Electrical

Electrical assistant that can install cable trays and cabling, must reside close to Durbanville area.


View Job
Electrical Assistant

Durban: Fleet Sales Executive posted by Intercon Recruitment

Fleet Sales Executive Durban, KZNA leading and high-performing dealership in Durban is currently seeking an experienced and results-driven Fleet Sales...


View Job
Fleet Sales Executive

Cape Town City Centre: Travel Desk Administrator / Administrative Assistant – Durbanville

Job Specification A well-rounded person with a high EQ level and must be able to deal with complexity in a fast-paced environment The person needs to be a...


View Job
Travel Desk Administrator / Administrative Assistant – Durbanville

Durban: Senior Process Engineer

Job Description Elimination of Waste Assist in and support the ME plans on existing and new production lines. Support ME Improvement objectives by analysis...


View Job
Senior Process Engineer

Durban: New Vehicle Sales Executive posted by Intercon Recruitment

New Vehicle Sales Executive - DurbanA high-performing and well-established dealership in Durban is currently seeking a dynamic and experienced New Vehicle...


View Job
New Vehicle Sales Executive

Durban: HR Consultant

Source and short-list appropriate applicants for vacancies as identified for department managers Guide the applicant selection process Assist disciplinary...


View Job
HR Consultant

Durban: MOULDER – REFINERY

MOULDER with REFINERY Industry experience - Compulsory Minimum 3-4 Years working experience in a Foundry Clean Criminal record - will be verified


View Job
MOULDER – REFINERY

Durban Central: Administration Clerk posted by Victor Crescent

Our company is looking for a focused, industrious, and likable candidate to fill a vacant administrative clerk position. As an administrative clerk, you...


View Job
Administration Clerk

Durban Central: Driver Code 10 posted by Victor Crescent

The driver is responsible for providing a safe and reliable bus service and for transporting employees to and from work. He or she will also perform work...


View Job
Driver Code 10

Durban Central: Driver Code 14 posted by Victor Crescent

Victor Crescent Construction is currently looking for a driver code 14 to work on our warehouse distribution.MINIMUM REQUIREMENTSUse the Location and...


View Job
Driver Code 14

Durban Central: Forklift Operator posted by Victor Crescent

Forklift operators are responsible for operating and managing industrial trucks to load and unload materials and deliveries and move them to and from...


View Job
Forklift Operator

Durban: PRODUCTION PLANNER

PRODUCTION PLANNER - MUST have FMCG Industry experience within a SUPPLY CHAIN environment Qualification in SUPPLY CHAIN MANAGEMENT LOGISTICS Minimum 3...


View Job
PRODUCTION PLANNER

Share this to someone who needs a job:
Posted in Jobs in Durban, Jobs in Kwazulu-Natal, Jobs in South Africa, Other IT/Computer

More Jobs in Your Area