Menu Close

South Africa: Senior Data Engineer (Kafka, Python, Elasticsearch) (Cpt Remote) posted by Datafin

Date 2024-11-24
Job Title Senior Data Engineer (Kafka, Python, Elasticsearch) (Cpt Remote)
Employer Datafin
More Information
Salary 0
Category It Computer
Location South Africa
/ South Africa

Job Summary

Senior Data Engineer (Kafka, Python, Elasticsearch) (CPT Remote)IT – Analyst, Data Management
Cape Town – Western Cape ~ Remote

ENVIRONMENT:
AN award-winning leader in contact centre AI software seeks a passionate Data Engineer with expertise in Kafka pipelines and a thorough understanding of Elastic, looking to contribute to cutting-edge technology and make a difference in the Financial Services industry. You will design, implement, and maintain robust data pipelines; troubleshoot data pipeline and Elasticsearch issues while ensuring data infrastructure aligns with business needs. The ideal candidate will have proven experience in designing and implementing data pipelines including End-to-End Testing of analytics pipelines & managing and optimising Elasticsearch clusters, including performance tuning and scalability. You will also be proficient with Python, Scala or Java and DevOps.
DUTIES:
  • Design, implement, and maintain robust data pipelines, ensuring the efficient and reliable flow of data across systems.
  • Develop and maintain Elasticsearch clusters, fine-tuning them for high performance and scalability.
  • Collaborate with cross-functional teams to Extract, Transform, and Load (ETL) data into Elasticsearch for advanced analytics and search capabilities.
  • Troubleshoot data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting.
  • Participate in the design and development of data models and schemas to support business requirements.
  • Continuously monitor and optimise data pipeline and Elastic performance to meet growing data demands.
  • Collaborate with Data Scientists and Analysts to enable efficient data access and query performance.
  • Contribute to the evaluation and implementation of new technologies and tools that enhance Data Engineering capabilities.
  • Demonstrate strong analytical, problem-solving, and troubleshooting skills to address data-related challenges.
  • Collaborate effectively with team members and stakeholders to ensure data infrastructure aligns with business needs.
  • Embody the company values of playing to win, putting people over everything, driving results, pursuing knowledge, and working together.
  • Implement standards, conventions and best practices.
REQUIREMENTS:
  • Proven experience in designing and implementing data pipelines.
  • Experience with End-to-End Testing of analytics pipelines.
  • Expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability.
  • Strong proficiency with data extraction, transformation, and loading (ETL) processes.
  • Familiarity with data modeling and schema design for efficient data storage and retrieval.
  • Good programming and scripting skills using languages like Python, Scala, or Java.
  • Knowledge of DevOps and automation practices related to Data Engineering.
As a Data Engineer with a focus on Kafka pipelines and Elastic, you will work with the following technologies:
Data Pipelines:
    • Kafka / ksqlDB
    • Python
    • Redis
Data Storage and Analysis:
    • Elasticsearch, cluster management and optimisation
    • AWS S3
    • PostgreSQL
DevOps:
    • AWS
Advantageous
  • Experience with Data Engineering in an Agile / Scrum environment.
  • Familiarity with ksqlDB / Kafka or other stream processing frameworks.
  • Familiarity of Data Lakes and the querying thereof.
  • Experience with integrating Machine Learning models into data pipelines.
  • Familiarity with other data-related technologies and tools.
ATTRIBUTES:
  • Strong analytical and problem-solving abilities, with a keen attention to detail.
  • Excellent communication and collaboration skills to work effectively with cross-functional teams.
  • A commitment to staying up to date with the latest developments in Data Engineering and technology.
  • Alignment with company values and a dedication to driving positive change through data.
Apply for this Job

Senior Data Engineer (Kafka, Python, Elasticsearch) (Cpt Remote) position available in South Africa, South Africa. This job position was posted by Datafin. The job has been posted as a premium ad on 2024-11-24 at 16:05:52 in the It Computer category

Click Go Apply to apply online!

View Job  South Africa: Quality Control Manager - Food Industry Experience Required posted by Top Talent Professional Services

You might also like to look at the available jobs in the same area.

Apply directly on jobmail for this position
Some posts have contact information included with other instructions. All applicants are urged to read the instructions carefully.
We will list jobs at the bottom of each job, so feel free to scroll down and continue your search.

We do not accept any job applications and merely offer the service of daily jobs being mined and displayed on our job portal.
Each job post has a link to take you directly to the original post to apply on their website or the original website where the post came from.


Here are some more related positions


Cape Town City Centre: Senior Data Scientist

We are seeking a highly skilled Senior Data Scientist to join our team and drive advanced data analytics, machine learning initiatives, and data-driven decision-making processes. The ideal candidate will have extensive experience in Python programming , wi


View Job
Senior Data Scientist

Johannesburg: Senior Data Analyst

Responsibilities - Interpret data, analyse results using statistical techniques and provide ongoing reports. - Provide quality assurance of imported data, working with quality assurance analysts if necessary. - Commissioning and decommissioning of data set


View Job
Senior Data Analyst

Cape Town City Centre: Senior Data And Vehicle Support Engineer (Telematics) Cape Town

Senior Data and Vehicle Support Engineer (Telematics) Cape Town Minimum requirements and experience: BEng or MEng (Electronic or Mechatronic Engineering) Minimum of 5 years working experience in telematics systems and -components including data managemen


View Job
Senior Data And Vehicle Support Engineer (Telematics) Cape Town

Centurion: Senior Data And Vehicle Support Engineer (Telematics) Centurion

Senior Data and Vehicle Support Engineer (Telematics) Centurion Minimum requirements and experience: BEng or MEng (Electronic or Mechatronic Engineering) Minimum of 5 years working experience in telematics systems and -components including data managemen


View Job
Senior Data And Vehicle Support Engineer (Telematics) Centurion

Johannesburg: Senior Data Scientist – Johannesburg – R700k To R900k Per Annum

A financial house with leading business knowledge, industry experience and a variety of services to help businesses make an impact and a workforce made up of the industrys greatest minds, who continue to shape the future by delivering real, measurable resu


View Job
Senior Data Scientist – Johannesburg – R700k To R900k Per Annum

Johannesburg: Senior Database Administrator

Job & Company Description: As a Senior Database Administrator you will play a crucial role in ensuring the smooth operation of our organisations critical database systems. They will be responsible for architecting, implementing and optimising high performa


View Job
Senior Database Administrator

Cape Town City Centre: Senior Data Engineer

Key Responsibilities: Automate workflows and optimise data delivery. Build and maintain efficient CI/CD processes. Design large, complex data sets for business needs. Maintain data quality, governance, and infrastructure. Tech Stack: Python, PySpark, Palan

View Job  South Africa: Food And Beverage Manager posted by Career Custodians

View Job
Senior Data Engineer

Port Elizabeth: Senior Data Scientist – Port Elizabeth – 6-Month Contract – R70k To R130k Per Month

Join a company filled with like-minded individuals who deliver quality solution delivery services and prioritize flexibility in work arrangements, embracing both remote and hybrid models to cater to our staff``s preferences as a Senior Data Scientist you w


View Job
Senior Data Scientist – Port Elizabeth – 6-Month Contract – R70k To R130k Per Month

Johannesburg: Senior Data Scientist – Johannesburg – R1mil To R1.3Mil Per Annum

An African financial services organisation that offers a broad spectrum of financial solutions and offers great culture, leadership, learning opportunities, challenges, rewards and recognition to its employees seeks a Senior Data Scientist who will be acco


View Job
Senior Data Scientist – Johannesburg – R1mil To R1.3Mil Per Annum

Cape Town City Centre: Senior Data Engineer

We are looking for a Senior Data Engineer to support and manage BI analytics solutions. In this role, you should have a background in data and business intelligence. You should be analytical; an excellent communicator and a problem-solving aptitude is need


View Job
Senior Data Engineer

Johannesburg: Senior Data Engineer – Johannesburg – R750k To R950k Per Annum

A registered Financial Service Provider with an undivided team of diverse thinkers and doers who believe in leading through technology and pushing past their limits with innovation at its core is seeking a dynamic Senior Data Engineer to execute data engin


View Job
Senior Data Engineer – Johannesburg – R750k To R950k Per Annum

Cape Town City Centre: Senior Data Analyst

Key Responsibilities: Design and Development: Develop and maintain advanced analytics applications and data systems. Own key components of data solutions, ensuring they meet the needs of the business. Data Systems Implementation: Lead the development of da


View Job
Senior Data Analyst

Cape Town City Centre: Senior Data Analyst

Responsibilities: Design, develop and maintain analytics applications and systems Develop and implement data collection systems, data analytics, data archiving and other strategies Design and manage centralised data warehouse, defining data security measur


View Job
Senior Data Analyst

Johannesburg: Intermediate/ Senior Data Scientist – Johannesburg – R800k To R1mil Per Annum

A registered Financial Service Provider with an undivided team of diverse thinkers and doers who believe in leading through technology and pushing past their limits with innovation at its core is seeking a dynamic Intermediate/ Senior Data Scientist ready


View Job
Intermediate/ Senior Data Scientist – Johannesburg – R800k To R1mil Per Annum

Cape Town City Centre: Senior Data Analyst

Key Responsibilities: Design, develop, and maintain analytics applications and centralised data warehouses. Analyse and interpret trends in complex datasets, offering actionable insights. Ensure compliance with data security, privacy, and quality standards


View Job
Senior Data Analyst

Johannesburg: Senior Data Engineer (Sas, Aws) – Johannesburg – R850k To R1.1Mil Per Annum

Are you a Data Engineer with a passion for driving, designing and building scalable ETL systems for a big data warehouse to implement robust & trustworthy data to support high performing, ML algorithms, predictive models and support real-time data visualis


View Job
Senior Data Engineer (Sas, Aws) – Johannesburg – R850k To R1.1Mil Per Annum

Cape Town: Senior Data Scientist posted by PBT Group

We are seeking a highly skilled Senior Data Scientist to join our team and drive advanced data analytics, machine learning initiatives, and data-driven decision-making processes. The ideal candidate will have extensive experience in Python programming, wit

View Job  Gauteng: Medical Waste Treatment Plant Supervisor

View Job
Senior Data Scientist

Cape Town: Senior Data Engineer : Ingestions Squad (Group Data & Ml Platform Tribe) posted by Progressive IT Resourcing

Our ClientCompany Group Technology is responsible for the provision of a digitally enabled technology service as a group COE, drive business and transformation and provide group-wide digital and data architecture. We operate the various technology platform


View Job
Senior Data Engineer

Johannesburg: Senior Data Engineer posted by Deka Minas (Pty) Ltd

We are excited to introduce an exceptional opportunity for a Senior Data Engineer! In this pivotal role, you will spearhead the development and optimization of our data engineering solutions, focusing on modernizing our data warehousing and processing stra


View Job
Senior Data Engineer

Pretoria: Senior Manager: Digital And Data posted by People Dimension

Job Description/Duties: Formulation of a digital strategy Manage third-party IT providers and the development of data architecture. Manage and coordinate the development of bespoke systems, digital channels and other software development. Ensure solutions


View Job
Senior Manager

Johannesburg: Senior Data Analyst posted by Isilumko Staffing

Job description:The key accountabilities for the data analyst includes but is not limited to:Data collection and cleansing.Data Analysis.Reporting and visualisation.Data modelling and forecasting.Data interpretation and insights.Data quality and governance


View Job
Senior Data Analyst

Sandton: Senior Data Analyst (6 – 7 Month Contract) posted by Isilumko Staffing

Urgently looking for an experienced Senior Data Analyst to join a global energy and chemical client in Sandton, on a 6-7 month contract. Responsibilities:Data analysis and interpretation of complex dataInvolved with the decision making processOffer meaning


View Job
Senior Data Analyst (6 – 7 Month Contract)

Johannesburg: Senior Data Analyst posted by Isilumko Staffing

We are seeking highly skilled and experienced Data Analysts to join our dynamic team. As a Data Analyst, you will play a pivotal role in collecting, analyzing, and interpreting large datasets to derive actionable insights that contribute to strategic decis


View Job
Senior Data Analyst

Johannesburg: Group Senior Manager : Digital Marketing Platforms, Services & Data posted by Isilumko Staffing

One of the TOP banks in S.A. is looking for an experienced data driven, analytical, system orientated platform FUndi for their digital marketing team, for a 6-8 month hybrid opportunity, towards Rosebank. Manage 4 direct reports You will be responsible for


View Job
Group Senior Manager

Johannesburg: Senior Manager: Data-Driven Marketing (6 Months) posted by Isilumko Staffing

One of the TOP banks in South Africa is looking for a Senior Marketing Manager. You will be responsible for managing programmes, managing technologies, etc. Requirements:Proficient in digital marketing.Digital media knowledge.Proactive.A self-starter. Se


View Job
Senior Manager

Error making API request.

Share this to someone who needs a job:
Posted in Jobs in South Africa

More Jobs in Your Area