Menu Close

South Africa: Senior Data Engineer (Kafka, Python, Elasticsearch) (Cpt Remote) posted by Datafin

Date 2024-10-13
Job Title Senior Data Engineer (Kafka, Python, Elasticsearch) (Cpt Remote)
Employer Datafin
More Information
Salary 0
Category It Computer
Location South Africa
/ South Africa

Job Summary

Senior Data Engineer (Kafka, Python, Elasticsearch) (CPT Remote)IT – Analyst, Data Management
Cape Town – Western Cape ~ Remote

ENVIRONMENT:
AN award-winning leader in contact centre AI software seeks a passionate Data Engineer with expertise in Kafka pipelines and a thorough understanding of Elastic, looking to contribute to cutting-edge technology and make a difference in the Financial Services industry. You will design, implement, and maintain robust data pipelines; troubleshoot data pipeline and Elasticsearch issues while ensuring data infrastructure aligns with business needs. The ideal candidate will have proven experience in designing and implementing data pipelines including End-to-End Testing of analytics pipelines & managing and optimising Elasticsearch clusters, including performance tuning and scalability. You will also be proficient with Python, Scala or Java and DevOps.
DUTIES:
  • Design, implement, and maintain robust data pipelines, ensuring the efficient and reliable flow of data across systems.
  • Develop and maintain Elasticsearch clusters, fine-tuning them for high performance and scalability.
  • Collaborate with cross-functional teams to Extract, Transform, and Load (ETL) data into Elasticsearch for advanced analytics and search capabilities.
  • Troubleshoot data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting.
  • Participate in the design and development of data models and schemas to support business requirements.
  • Continuously monitor and optimise data pipeline and Elastic performance to meet growing data demands.
  • Collaborate with Data Scientists and Analysts to enable efficient data access and query performance.
  • Contribute to the evaluation and implementation of new technologies and tools that enhance Data Engineering capabilities.
  • Demonstrate strong analytical, problem-solving, and troubleshooting skills to address data-related challenges.
  • Collaborate effectively with team members and stakeholders to ensure data infrastructure aligns with business needs.
  • Embody the company values of playing to win, putting people over everything, driving results, pursuing knowledge, and working together.
  • Implement standards, conventions and best practices.
REQUIREMENTS:
  • Proven experience in designing and implementing data pipelines.
  • Experience with End-to-End Testing of analytics pipelines.
  • Expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability.
  • Strong proficiency with data extraction, transformation, and loading (ETL) processes.
  • Familiarity with data modeling and schema design for efficient data storage and retrieval.
  • Good programming and scripting skills using languages like Python, Scala, or Java.
  • Knowledge of DevOps and automation practices related to Data Engineering.
As a Data Engineer with a focus on Kafka pipelines and Elastic, you will work with the following technologies:
Data Pipelines:
    • Kafka / ksqlDB
    • Python
    • Redis
Data Storage and Analysis:
    • Elasticsearch, cluster management and optimisation
    • AWS S3
    • PostgreSQL
DevOps:
    • AWS
Advantageous
  • Experience with Data Engineering in an Agile / Scrum environment.
  • Familiarity with ksqlDB / Kafka or other stream processing frameworks.
  • Familiarity of Data Lakes and the querying thereof.
  • Experience with integrating Machine Learning models into data pipelines.
  • Familiarity with other data-related technologies and tools.
ATTRIBUTES:
  • Strong analytical and problem-solving abilities, with a keen attention to detail.
  • Excellent communication and collaboration skills to work effectively with cross-functional teams.
  • A commitment to staying up to date with the latest developments in Data Engineering and technology.
  • Alignment with company values and a dedication to driving positive change through data.
Apply for this Job

Senior Data Engineer (Kafka, Python, Elasticsearch) (Cpt Remote) position available in South Africa, South Africa. This job position was posted by Datafin. The job has been posted as a premium ad on 2024-10-13 at 16:09:24 in the It Computer category

Click Go Apply to apply online!

View Job  South Africa: Business Developer - Sandton posted by Professional Career Services

You might also like to look at the available jobs in the same area.

Apply directly on jobmail for this position
Some posts have contact information included with other instructions. All applicants are urged to read the instructions carefully.
We will list jobs at the bottom of each job, so feel free to scroll down and continue your search.

We do not accept any job applications and merely offer the service of daily jobs being mined and displayed on our job portal.
Each job post has a link to take you directly to the original post to apply on their website or the original website where the post came from.


Here are some more related positions


City Bowl: Senior Data Analyst/Scientist posted by Lebo Business Consulting (Pty) Ltd

Manage performance of data analyst/operator team; Play a leading role in the designing and documenting of operating procedures; Conduct, monitor and calibrate astronomical observations and use appropriate computing resources towards successful operation of


View Job
Senior Data Analyst/Scientist

Pretoria: Senior Manager: Digital And Data

Job Description/Duties: · Formulation of a digital strategy · Manage third-party IT providers and the development of data architecture. · Manage and coordinate the development of bespoke systems, digital channels and other software development. · Ensure so


View Job
Senior Manager

South Africa: Senior Data Scientist – Cape Town – 6-Month Contract – R70k To R130k Per Month

Join a company filled with like-minded individuals who deliver quality solution delivery services and prioritize flexibility in work arrangements, embracing both remote and hybrid models to cater to our staff``s preferences as a Senior Data Scientist you w


View Job
Senior Data Scientist – Cape Town – 6-Month Contract – R70k To R130k Per Month

South Africa: Senior Data Engineer – Remote – R800k To R1 Mil Per Annum

Join a market leader in compliant call recording solutions and beyond capturing every interaction, with analytics platform extracts, processes and delivers insights from millions of data points across diverse communication channels the analytics team is lo


View Job
Senior Data Engineer – Remote – R800k To R1 Mil Per Annum

Cape Town City Centre: Senior Data Engineer

Responsibilities: Design and implement scalable data pipelines using Apache Spark, Scala, and Java Develop and maintain data architectures for batch and real-time processing Collaborate with cross-functional teams to identify and prioritize project require


View Job
Senior Data Engineer

Centurion: Senior Database Administrator

Key Responsibilities : Requirements Gathering : Collaborate with business stakeholders to collect, document, and review database requirements. Database Design & Development : Architect, develop, and implement scalable, reliable databases that meet business


View Job
Senior Database Administrator

Cape Town City Centre: Senior Data And Vehicle Support Engineer (Telematics) Cape Town

Senior Data and Vehicle Support Engineer (Telematics) Cape Town Minimum requirements and experience: BEng or MEng (Electronic or Mechatronic Engineering) Minimum of 5 years working experience in telematics systems and -components including data management

View Job  South Africa: Driver (Code 10) X3

View Job
Senior Data And Vehicle Support Engineer (Telematics) Cape Town

Centurion: Senior Data And Vehicle Support Engineer (Telematics) Centurion

Senior Data and Vehicle Support Engineer (Telematics) Centurion Minimum requirements and experience: BEng or MEng (Electronic or Mechatronic Engineering) Minimum of 5 years working experience in telematics systems and -components including data management


View Job
Senior Data And Vehicle Support Engineer (Telematics) Centurion

Johannesburg: Senior Data Scientist – Johannesburg – Renewable Contract– Up To R100k Per Month

Join a company filled with like-minded individuals who deliver quality solution delivery services and prioritize flexibility in work arrangements, embracing both remote and hybrid models to cater to our staff``s preferences as a Senior Data Scientist you w


View Job
Senior Data Scientist – Johannesburg – Renewable Contract– Up To R100k Per Month

Johannesburg: Senior Sql Data Engineer

As a Specialist Recruitment Consultant within the IT industry, my clients and I are constantly on the lookout for accomplished Senior SQL Data Engineer with strong academics, robust project experience and technical knowledge. Requirements: Completed qualif


View Job
Senior Sql Data Engineer

Johannesburg: Senior Data Scientist

Qualifications and Experience required 5-8 years of experience as a Data Scientist with expertise in machine learning and AI technologies. Proficient in Python and experience with deep learning frameworks such as TensorFlow. Hands-on experience with machin


View Job
Senior Data Scientist

South Africa: Senior Data Scientist – Cape Town – R1.2Mil To R1.4Mil Per Annum

A dynamic, industrial holding company dedicated to investing in, growing and starting businesses a global footprint in diverse industries including manufacturing, distribution, technology, big data, engineering and energy efficiency seek a Senior Data Scie


View Job
Senior Data Scientist – Cape Town – R1.2Mil To R1.4Mil Per Annum

Cape Town City Centre: Senior Data Scientist / Ai Developer (Ch1051) – Fully Remote

Our client is a leading software services and consulting company focusing on delivering innovative solutions with a strategic emphasis on generative AI. They are looking for an experienced Senior Data Scientist / AI Developer / Data Engineer to join their


View Job
Senior Data Scientist / Ai Developer (Ch1051) – Fully Remote

Johannesburg: Senior Data Analyst

Requirements: Understanding and application of Big Data Design and assist in the optimization of Data Warehouse/Data Lake and Reporting Development, testing and administration of BI models(tabular, SSIS, SSAS & Visual Studio) Proficiency in data manipulati


View Job
Senior Data Analyst

Cape Town City Centre: Senior Data Engineer : Ingestions Squad (Group Data & Ml Platform Tribe)

Our Client Company Group Technology is responsible for the provision of a digitally enabled technology service as a group COE, drive business and transformation and provide group-wide digital and data architecture. We operate the various technology platfor


View Job
Senior Data Engineer

Johannesburg: Senior Data Analyst

We are seeking a highly skilled Data Management Specialist to join our dynamic team. The successful candidate will play a critical role in linking external data to master data, ensuring the accuracy of inbound data through analysis and troubleshooting. You


View Job
Senior Data Analyst

Johannesburg: Senior Data Scientist

Duties: Optimisation and programming as a skill set on all platforms Understanding all the latest AI and ML Action planner Experience handling high level projects Has developed software from the beginning Job Experience & Skills Required: 5 years experienc

View Job  South Africa: Professional Services Consultant / Onboarding Consultant

View Job
Senior Data Scientist

Johannesburg: Senior Data Engineer posted by Deka Minas (Pty) Ltd

We are excited to introduce an exceptional opportunity for a Senior Data Engineer! In this pivotal role, you will spearhead the development and optimization of our data engineering solutions, focusing on modernizing our data warehousing and processing stra


View Job
Senior Data Engineer

Pretoria: Senior Manager: Digital And Data posted by People Dimension

Job Description/Duties: Formulation of a digital strategy Manage third-party IT providers and the development of data architecture. Manage and coordinate the development of bespoke systems, digital channels and other software development. Ensure solutions


View Job
Senior Manager

Cape Town: Senior Data Analyst posted by PBT Group

PBT Group is seeking a Senior Data Analyst with at least 3 years of relevant experience within the insurance or financial services industry. The successful candidate will support and maintain the data warehouse in line with the data model and metadata repo


View Job
Senior Data Analyst

Stellenbosch: Senior Decision (Data) Scientist (Ch1033) posted by Capital H Staffing and Advisory Solutions

Our client in the banking industry is looking for experienced Senior Decision Scientists.The purpose of the role is delivering business solutions via a collaborative approach involving mathematical formulae, business tactics, technological applications, an


View Job
Senior Decision (Data) Scientist (Ch1033)

Centurion: Senior Data Engineer posted by CF & I Consulting

The Data Engineer will have a knack for data analysis, data manipulation and data modelling and will be responsible for understanding and driving the overall technical vision and planning of a client`s organisation and translating business needs into techn


View Job
Senior Data Engineer

Cape Town: Senior Data Scientist posted by PBT Group

PBT Group is seeking a Senior Data Scientist to spearhead the development of retail Analytical Data Assets (ADA) and associated strategies for key `Use Cases` using data analysis, predictive modeling, simulation, and optimization techniques to identify tre


View Job
Senior Data Scientist

East London: Senior Data Specialist (Permanent) posted by Profile Personnel

Main purpose of the jobOwnership of Outsourced Credit Controls data and processes including, but not limited to, development, implementation and maintenance of operational and reporting processes. Engaging with business to ensure understanding and complian


View Job
Senior Data Specialist (Permanent)

Johannesburg: Senior Data Analyst posted by Isilumko Staffing

Job description:The key accountabilities for the data analyst includes but is not limited to:Data collection and cleansing.Data Analysis.Reporting and visualisation.Data modelling and forecasting.Data interpretation and insights.Data quality and governance


View Job
Senior Data Analyst

Error making API request.

Share this to someone who needs a job:
Posted in Jobs in South Africa

More Jobs in Your Area