Gauteng: Data Engineer/Scientist (Advanced) 1210 posted by Opensource
Posted on 2025-06-16 00:00:00
| Employer | Opensource |
|---|---|
| Category | International Beauty/Wellness |
| Location | Gauteng / Gauteng |
Job Summary
- Proficiency in Python 3.x for data processing and automation.
- Experience with AWS Glue for ETL processes.
- Strong knowledge of AWS Athena for querying large datasets.
- Hands-on experience with AWS Lambda for serverless computing.
- Familiarity with AWS EC2 for scalable computing resources.
- Expertise in AWS CloudWatch for monitoring and logging.
- Proficiency in working with PostgreSQL RDS for database management.
- Experience with AWS QuickSight for data visualization and reporting.
- Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs).
- Proficiency in Git and GitHub for version control and collaboration, including experience with GitHub CI/CD pipelines.
- Any additional responsibilities assigned in the Agile Working Model (AWM) Charter
Advantageous skills requirements:
- Extensive experience with Terraform for infrastructure as code, including creating, managing, and optimizing AWS resources.
- Experience with other AWS services such as S3, Kinesis, and DynamoDB.
- Knowledge of data formats such as Parquet, AVRO, JSON, and CSV.
- Experience with Docker for containerization.
- Understanding of Big Data technologies and frameworks.
- Familiarity with Agile working models and tools like JIRA and Confluence.
- Experience with data quality tools such as Great Expectations.
- Knowledge of REST API development and integration.
- Strong analytical skills for troubleshooting and optimizing data pipelines.
- Experience in developing technical documentation and artefacts.
What will be your role and responsibilities?
- Design, develop, and maintain data ingestion pipelines for BMW’s connected fleet, focusing on Call Detail Records (CDRs).
- Utilize AWS Glue for ETL processes to transform and load data into the analytics platform.
- Implement efficient querying solutions using AWS Athena.
- Develop serverless applications and workflows using AWS Lambda.
- Monitor and optimize data pipelines using AWS CloudWatch.
- Manage and maintain PostgreSQL RDS databases for data storage and retrieval.
- Create interactive dashboards and reports using AWS QuickSight.
- Leverage Terraform extensively to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability.
- Use Git and GitHub for version control, collaboration, and maintaining code repositories.
- Implement and manage GitHub CI/CD pipelines to automate testing, deployment, and delivery processes.
- Ensure data security and compliance with BMW’s information classification requirements.
- Stay updated with the latest data engineering tools, technologies, and industry trends.
- Identify opportunities for process improvements and automation to enhance pipeline efficiency.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Develop and maintain technical documentation for data engineering processes and solutions.
Apply now!!! send CV to *****@*****.co.za
Click Go Apply to apply online!
Get Similar Job Alerts
More jobs in Gauteng
More jobs in Gauteng
Browse International Beauty/Wellness Jobs
Browse Opensource Jobs
Apply directly for this position. Please read all instructions carefully.
We do not process job applications; we simply aggregate and display job listings.
More related positions
Gauteng: Sales Representative posted by UNOUNO
Plan and schedule visits to all existing customersIdentify opportunities to increase sales to existing customers by more sales of current products andor...
View Job
Sales Representative
Gauteng: Turner for Centre Lathe – Must be qualified posted by Thynder Holdings
Looking for a Centre Lathe turner with experience,Position based in Germiston area.
View Job
Turner for Centre Lathe – Must be qualified
Gauteng: Retail Operations Manager posted by 3at1 Franchising
About Us31 Business Centre Franchising is a leading national franchise retailer in South Africa. We are a dynamic, technology-driven company committed to...
View Job
Retail Operations Manager
Gauteng: Senior Retail Store Buyer posted by African Premier Traders
The Senior Retail Store Buyer plays a critical role in driving the profitability and success of the store by identifying, selecting, and purchasing products...
View Job
Senior Retail Store Buyer
Gauteng: Vending Machine Attendant – Urgently Needed posted by Always Open
Job Opportunity Part-Time Vending Machine AttendantNo previous experience required.We are looking for a reliable and hardworking Vending Machine Attendant...
View Job
Vending Machine Attendant – Urgently Needed
Gauteng: Millwright – Shift International posted by Crystal Contracting SA
Excellent Shift Millwright required with TT4 days on 4 days off. International companyDefinite PLC programming exp, pneumatics 2 Bonuses per year 13th...
View Job
Millwright – Shift International
Gauteng: Sales Rep – Urgent needed posted by Evolution fuels
We are looking for an experienced Salesperson to join our team. The ideal candidate must meet the following criteriaSales Experience is essentialOwn...
View Job
Sales Rep – Urgent needed
Gauteng: LLB Graduate – Available Immediately posted by Solver Property Services
LLB Graduate, post- article with Sectional Title experienceExperience in CSOSDebt collection process etc.
View Job
LLB Graduate – Available Immediately
Gauteng: Sales Representative posted by Emthunzini Finacial Services
Exciting Career Opportunity Life Funeral Insurance Sales ConsultantAre you ready to take your career to new heights? We are looking for motivated and...
View Job
Sales Representative
Gauteng: Defended Banking Litigation Attorney posted by HR Guy
Well established Legal Firm requires the services of a Defended Banking Litigation Attorney to join their team.Experience requiredMinimum of 7 years post...
View Job
Defended Banking Litigation Attorney
Gauteng: Clinical Trial Nurse – Urgently needed posted by Medical Careers
A renowned oncology centre is seeking an Afrikaans and English-speaking registered nurse as a Clinical Trial Nurse to support the delivery of sponsored...
View Job
Clinical Trial Nurse – Urgently needed
Gauteng: Regulatory & Stakeholder Officer – Urgently needed posted by Medical Careers
Were currently looking for a Pharmacist with strong regulatory marketing code knowledge to take on a dynamic role as a Regulatory and Stakeholder Officer...
View Job
Regulatory & Stakeholder Officer – Urgently needed
Gauteng: Sales Agents – Urgently posted by World E Hailing TA My Rides PTY Limited
Job Opportunity Sales Agents for E-Hailing CompanyWe are seeking highly motivated and results-driven Sales Agents to join our team at a leading South...
View Job
Sales Agents – Urgently
Gauteng: Plate Layer – Urgently needed posted by African Black Bear Holdings (Pty) Ltd
Track Master Certificate Accredited by Transnet and Experience as a qualified Track Master.
View Job
Plate Layer – Urgently needed
Gauteng: Drivers – Urgently Needed posted by MCM Total Construction and Engineering Management
Urgently looking for experienced drivers with Professional Driving Permit PrDP. Code EC 14 with PrDP as well as Code C Code 10 with PrDP. Successful...
View Job
Drivers – Urgently Needed
Gauteng: Assemblers X3 – Must have experience in woodwork posted by Linear Building Projects
Construct and assemble cabinet components, making sure that all pieces fit the dimensions specified by the supervisor Verify and double check all...
View Job
Assemblers X3 – Must have experience in woodwork
Gauteng: Safety Officer – Urgently Needed posted by African Black Bear Holdings (Pty) Ltd
Safety OfficerRequirements1. National Diploma in Safety Management National Diploma in Environmental Health.2. Valid registration with SACPCMP as a Safety...
View Job
Safety Officer – Urgently Needed
Gauteng: Specialized Investigator – Covert Security & Intelligence – Urgent posted by Alwinco
We are seeking a highly skilled and discreet investigator with strong analytical abilities, excellent observational skills, and a commitment to ethical...
View Job
Specialized Investigator – Covert Security & Intelligence – Urgent
Gauteng: Sales Representative posted by RR Industrial
RR Industrial has a vacancy for a sales representative, Person must be able to meet the set criteria, Should be able to take own initiative, Must have own...
View Job
Sales Representative
Gauteng: MATHS TEACHER posted by The Teacher Network
MATHS TEACHER Grades 8-9MayJune 2025 Our client, a well-established independent school on the West Rand Johannesburg, needs a Grade 8-9 teacher as soon as...
View Job
MATHS TEACHER
Gauteng: Truck driver 18m3 tipper – Tipper truck driver posted by AQUALITY PLUMBERS
We are looking for a Tipper truck driver for the Gauteng area.
View Job
Truck driver 18m3 tipper – Tipper truck driver
Gauteng: Call Centre Agent posted by Newtons Incorporated
We are an attorneys firm in Brooklyn, Pretoria looking for 2 experienced call centre agents to join our team. The Candidate will need to understand the...
View Job
Call Centre Agent
Gauteng: Company Reps posted by Royal Distributors Pty Ltd
We require 6 company representatives to call on retailers and wholesalers in Gauteng, Free State, Northern Cape, North West, Limpopo and Mpumlanga.Our...
View Job
Company Reps
Gauteng: Millwright posted by Crystal Contracting
Shift MillwrightInternational CompanyKrugersdorp GautengN4 Millwright Trade TestExp Siemens PLC Programming a mustPneumatics4 Days on4 Days offMust be able...
View Job
Millwright
Gauteng: Learning Assistant posted by GLOBAL CONSULTING
Global Consulting requires candidates for the position of Learning Assistant. Candidates should email a detailed CV.
View Job
Learning Assistant