Date | 2025-01-18 |
---|---|
Job Title | Senior Data Engineer |
Employer | More Information |
Salary | Monthly |
Category | It Computing Software |
Location |
eThekwini / Durban |
Job Summary
This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills Design, develop, and maintain the end-to-end technical aspects (hardware – on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers. Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks Optimize data performance, scalability, and security Provide technical guidance and support to data analysts and research scientists. Design data integrations and data quality frameworks Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture Document and effectively communicate data engineering processes and solutions. Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments. Minimum Qualifications: Bachelor`s degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field Industry recognized IT related certification and technology qualification such as Databases and Data related certifications. This is a technical role so a strong focus needs to be on technical skills and experience Minimum Experience: 7 years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies. Knowledge and Abilities: Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc. Experience of Big Data technologies such as Hadoop, Spark and Hive Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow Experience working with containerization is advantageous Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure. Strong knowledge of data modeling, architecture, and governance principles Strong Linux Administration skills Programming skills in various languages advantageous Strong data security and compliancy experience Excellent communication, collaboration, and problem-solving skills Ability to work independently and as part of a cross-functional team Interest and enthusiasm for medical scientific research and its applications SUMMARY: There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc. You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.
Senior Data Engineer position available in eThekwini, Durban. This job position was posted by . The job has been posted on 2025-01-18 in the It Computing Software category
Click Go Apply to apply online!
You might also like to look at the available jobs in the same area.
Apply directly for this position
Some posts have contact information included with other instructions. All applicants are urged to read the instructions carefully.
We will list jobs at the bottom of each job, so feel free to scroll down and continue your search.
We do not accept any job applications and merely offer the service of daily jobs being mined and displayed on our job portal.
Each job post has a link to take you directly to the original post to apply on their website or the original website where the post came from.
Here are some more related positions
Johannesburg: Senior Data Analyst – Experienced posted by All Star Placements
We have a great position available for a Senior Data Analyst in the Woodmead area for a head office in the mining industry.Candidates will be expected to: Monthly Reports - Preparation of all reports required for mine month end processing. Monthly Manageme
View Job
Senior Data Analyst – Experienced
Cape Town City Centre: Senior Data Analyst
Key Responsibilities: Design and Development: Develop and maintain advanced analytics applications and data systems. Own key components of data solutions, ensuring they meet the needs of the business. Data Systems Implementation: Lead the development of da
View Job
Senior Data Analyst
Cape Town City Centre: Senior Software Engineer – Data, Group Data & Ml Platform Tribe
What will you do ? Our client is a newly created engineering organisation on a mission to empower and accelerate autonomous value creation through insights and ML across the group. The company main users are the data scientists and analysts who sit inside
View Job
Senior Software Engineer – Data, Group Data & Ml Platform Tribe
Cape Town City Centre: Senior Hubspot Crm & Data Specialist
Who are we? A fast-paced, innovation-driven company specialising in spend management. They develop cloud-based software that simplifies procurement processes for mid-sized companies, enabling them to manage, optimise, and analyse business spend effectively
View Job
Senior Hubspot Crm & Data Specialist
Cape Town City Centre: Senior Project Manager – Investment Data Projects
Job Description This progressive insurance company is on the lookout for an individual to join their projects department as a Senior Project Manager - Investment Data Projects. The suitable candidate will need to have a keen interest in working within the
View Job
Senior Project Manager – Investment Data Projects
Menlyn: Aiqx Data Scientist Specialist (Senior) 1785
Essential Skills Requirements: Do you have what it takes? Were looking for exceptional talent with: Strong expertise in Applied Machine Learning , Data Science , and Deep Learning . Proven experience building predictive models using frameworks like TensorF
View Job
Aiqx Data Scientist Specialist (Senior) 1785
Johannesburg: Senior Teradata Data Engineer – Contract – Sandton – R680 Per Hour
We are currently in search for a Data Engineer with 5 to 8 years experience in data engineering and strong backend and front-end development skills. You will be required to build data warehouse components on Teradata warehouse (3NF) and will have to comply
View Job
Senior Teradata Data Engineer – Contract – Sandton – R680 Per Hour
Cape Town: Senior Hubspot Crm & Data Specialist posted by Ad Talent Africa
Who are we?A fast-paced, innovation-driven company specialising in spend management. They develop cloud-based software that simplifies procurement processes for mid-sized companies, enabling them to manage, optimise, and analyse business spend effectively.
View Job
Senior Hubspot Crm & Data Specialist
Cape Town: Senior Data Engineer : Ingestions Squad (Group Data & Ml Platform Tribe) posted by Progressive IT Resourcing
Our ClientCompany Group Technology is responsible for the provision of a digitally enabled technology service as a group COE, drive business and transformation and provide group-wide digital and data architecture. We operate the various technology platform
View Job
Senior Data Engineer
Pretoria: Senior Manager: Digital And Data posted by People Dimension
Job Description/Duties: Formulation of a digital strategy Manage third-party IT providers and the development of data architecture. Manage and coordinate the development of bespoke systems, digital channels and other software development. Ensure solutions
View Job
Senior Manager
Midrand: Senior Manager Data Engineer posted by Deka Minas (Pty) Ltd
RESPONSIBILITIESCreate and execute a long-term strategy for leveraging data and analytics as a key business enabler and competitive differentiator across the business.Take responsibility for developing and delivering a key element of the organisation`s dat
View Job
Senior Manager Data Engineer
Johannesburg: Senior Manager: Data Science And Business Intelligence posted by Telebest
Our client has an opportunity available for a Senior Manager in the Data Science and Business Intelligence department based in Sandton.Requirements:Postgraduate degree in and Analytical or Qualitative Field.An MBA or additional financial qualification woul
View Job
Senior Manager
Johannesburg: Senior Data Analytics Specialist posted by IOCO
We are seeking a Senior Data Analytics Specialist with excellent problem-solving skills, attention to detail, and the ability to translate data insights into strategic business recommendations. You will be responsible for providing data-driven insights an
View Job
Senior Data Analytics Specialist
Durban: Senior Data Engineer posted by Carlysle Human Capital
This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutio
View Job
Senior Data Engineer
Cape Town: Senior Data Analyst posted by Goldman Tech
Job Description:We have an amazing opportunity for one of our clients in Bellville - Cape Town.We require a candidate with:• Diploma in Data analysis / Information sciences / IT• At least Matric• 2 or more years in data processing / cleaning
View Job
Senior Data Analyst
Cape Town: Senior Data Analyst posted by Goldman Tech
Job Description:We are seeking a highly skilled and experienced Senior Data Analyst to join one of our Financial Services clients. The ideal candidate will be responsible for analyzing complex data sets and providing valuable insights to inform our busines
View Job
Senior Data Analyst
South Africa: Senior Data Engineer posted by Goldman Tech
Job Description:We have an amazing opportunity for a Senior Data EngineerWe require a candidate with:• Degree in Data Science, Engineering, Computer Science or Math.• Cloud certifications advantageous.• 5-10 years of experience in a data eng
View Job
Senior Data Engineer
Johannesburg: Senior Data Analyst posted by Goldman Tech
Job Description:One of our clients in Johannesburg is searching for an innovative senior data analyst to join their team. We require a candidate with:• Post matric qualifications (Quantitative or IT subjects are advantageous• 3 + years working ex
View Job
Senior Data Analyst
Johannesburg: Senior Data Warehouse Developer posted by Goldman Tech
Job Description:We have an amazing opportunity for a Senior Data Warehouse Developer in Johannesburg!We require a candidate with:• BSc. Computer Sciences or National Diploma in IT• 5+ years experience in a data warehousing role• 4+ years wri
View Job
Senior Data Warehouse Developer
Cape Town: Senior Sql Database Administrator posted by Goldman Tech
Job Description:We have an amazing opportunity for a Senior SQL Database Administrator in Cape Town!We require a candidate with:• A relevant 3-year Information Technology Degree or Diploma• Minimum of 10 years` experience in a similar role •
View Job
Senior Sql Database Administrator
Cape Town: Senior Data Analyst posted by Goldman Tech
Job Description:We have an amazing opportunity for a Senior Data Analyst in Cape Town! We require a candidate with experience in:• MS Excel• SQL query writing• Big data reporting packages (Power BI, Qlikview, Tableau)Should you meet the requ
View Job
Senior Data Analyst
Johannesburg: Senior Data Warehouse/ Bi Developer posted by Goldman Tech
Job Description:We have an amazing opportunity for a Senior Data Warehouse/ BI Developer in Johannesburg!We require a candidate with experience in:• BI Toolsets• ETL Applications• T-SQL• Implementation of DW/BI environments• BI Too
View Job
Senior Data Warehouse/ Bi Developer
Cape Town: Senior Data Scientist posted by Goldman Tech
Job Description:We have an exciting client in Cape Town looking for a Senior Data Scientist.You will:Build segmentation and propensity modelsBuild real time systems that integrate with the rest of our stackBuild visualisations using R Shiny, DataStudio, Ta
View Job
Senior Data Scientist
Cape Town: Senior Data Scientist posted by Goldman Tech
Job Description:We have a client in Cape Town looking for a Senior Data Scientist.You will:Build state-of-the-art statistical modelsBuild segmentation and propensity modelsBuild real time systems that integrate with the rest of our stackBuild visualisation
View Job
Senior Data Scientist
Cape Town: Senior Database Administrator posted by Goldman Tech
Job Description:Seeking a Senior Database Administrator in Cape Town for a Well-Known IT Company!Requirements:• National Diploma in Information Technology or Equivalent• 8+ Years` experience as a Database Administrator • Recent experience wi
View Job
Senior Database Administrator
Email Alerts
Click the button below and submit your email to receive email alerts with the latest jobs.
This service is offered by Result Generation
We do not save your information, and your email is the only information we forward to the service.
If you wish to unsubscribe, please click here