| Date | 2024-12-19 |
|---|---|
| Job Title | Data Engineer (Fully Remote) |
| Employer | More Information |
| Salary | 1000 Annually |
| Category | It Computing Software |
| Location |
Gauteng / Johannesburg |
Job Summary
Data Architecture and Design – Data Modeling: o Create normalized and denormalized schemas (3NF, star, snowflake). o Design data lakes, warehouses, and marts optimized for analytical or transactional workloads. o Incorporate modern paradigms like data mesh, lakehouse, and delta architecture. – ETL/ELT Pipelines: o Develop end-to-end pipelines for extracting, transforming, and loading data. o Optimize pipelines for real-time and batch processing. – Metadata Management: o Implement data lineage, cataloging, and tagging for better discoverability and governance. Distributed Computing and Big Data Technologies – Proficiency with big data platforms: o Apache Spark (PySpark, Sparklyr). o Hadoop ecosystem (HDFS, Hive, MapReduce). o Apache Iceberg or Delta Lake for versioned data lake storage. – Manage large-scale, distributed datasets efficiently. – Utilize query engines like Presto, Trino, or Dremio for federated data access. Data Storage Systems – Expertise in working with different types of storage systems: o Relational Databases (RDBMS): SQL Server, PostgreSQL, MySQL, etc. o NoSQL Databases: MongoDB, Cassandra, DynamoDB. o Cloud Data Warehouses: Snowflake, Google BigQuery, Azure Synapse, AWS Redshift. o Object Storage: Amazon S3, Azure Blob Storage, Google Cloud Storage. – Optimize storage strategies for cost and performance: o Partitioning, bucketing, indexing, and compaction. Programming and Scripting – Advanced knowledge of programming languages: o Python (pandas, PySpark, SQL Alchemy). o SQL (window functions, CTEs, query optimization). o R (data wrangling, Sparklyr for data processing). o Java or Scala (for Spark and Hadoop customizations). – Proficiency in scripting for automation (e.g., Bash, PowerShell). Real-Time and Streaming Data – Expertise in real-time data processing: o Apache Kafka, Kinesis, Azure Event Hub for event streaming. o Apache Flink or Spark Streaming for real-time ETL. o Implement event-driven architectures using message queues. – Handle time-series data and process live feeds for real-time analytics. Cloud Platforms and Services – Experience with cloud environments: o AWS: Lambda, Glue, EMR, Redshift, S3, Athena. o Azure: Data Factory, Synapse, Data Lake, Databricks. o GCP: BigQuery, Dataflow, Dataproc. – Manage infrastructure-as-code (IaC) using tools like Terraform or CloudFormation. – Leverage cloud-native features like auto-scaling, serverless compute, and managed services. DevOps and Automation – Implement CI/CD pipelines for data workflows: o Tools: Jenkins, GitHub Actions, GitLab CI, Azure DevOps. – Monitor and automate tasks using orchestration tools: o Apache Airflow, Prefect, Dagster. o Managed services like AWS Step Functions or Azure Data Factory. – Automate resource provisioning using tools like Kubernetes or Docker. Data Governance, Security, and Compliance – Data Governance: o Implement role-based access control (RBAC) and attribute-based access control (ABAC). o Maintain master data and metadata consistency. – Security: o Apply encryption at rest and in transit. o Secure data pipelines with IAM roles, OAuth, or API keys. o Implement network security (e.g., firewalls, VPCs). – Compliance: o Ensure adherence to regulations like GDPR, CCPA, HIPAA, or SOC 2. o Track and document audit trails for data usage. Performance Optimization – Optimize query and pipeline performance: o Query tuning (partition pruning, caching, broadcast joins). o Reduce IO costs and bottlenecks with columnar formats like Parquet or ORC. o Use distributed computing patterns to parallelize workloads. – Implement incremental data processing to avoid full dataset reprocessing. Advanced Data Integration – Work with API-driven data integration: o Consume and build REST/GraphQL APIs. o Implement integrations with SaaS platforms (e.g., Salesforce, Twilio, Google Ads). – Integrate disparate systems using ETL/ELT tools like: o Informatica, Talend, dbt (data build tool), or Azure Data Factory. Data Analytics and Machine Learning Integration – Enable data science workflows by preparing data for ML: o Feature engineering, data cleaning, and transformations. – Integrate machine learning pipelines: o Use Spark MLlib, TensorFlow, or scikit-learn in ETL pipelines. – Automate scoring and prediction serving using ML models. Monitoring and Observability – Set up monitoring for data pipelines: o Tools: Prometheus, Grafana, or ELK stack. o Create alerts for SLA breaches or job failures. – Track pipeline and job health with detailed logs and metrics. Business and Communication Skills – Translate complex technical concepts into business terms. – Collaborate with stakeholders to define data requirements and SLAs. – Design data systems that align with business goals and use cases. Continuous Learning and Adaptability – Stay updated with the latest trends and tools in data engineering: o E.g., Data mesh architecture, Fabric, and AI-integrated data workflows. – Actively engage in learning through online courses, certifications, and community contributions: o Certifications like Databricks Certified Data Engineer, AWS Data Analytics Specialty, or Google Professional Data Engineer.
Data Engineer (Fully Remote) position available in Gauteng, Johannesburg. This job position was posted by . The job has been posted on 2024-12-19 in the It Computing Software category
Click Go Apply to apply online!
You might also like to look at the available jobs in the same area.
Apply directly for this position
Some posts have contact information included with other instructions. All applicants are urged to read the instructions carefully.
We will list jobs at the bottom of each job, so feel free to scroll down and continue your search.
We do not accept any job applications and merely offer the service of daily jobs being mined and displayed on our job portal.
Each job post has a link to take you directly to the original post to apply on their website or the original website where the post came from.
Here are some more related positions
Pretoria North: Software Engineer – Sql Database Specialist posted by Sastel Packaging
Development and maintenance of inhouse database. Must have qualifications and experience in MySQL, Python and C#. Software Engineer - Sql Database Specialist position available in Gauteng, Pretoria North. This job position was posted by Sastel Packaging. T
View Job
Software Engineer – Sql Database Specialist
Centurion: Data Science Engineer – Centurion posted by MJM RECRUITMENT
We are hiring:Data science engineer x 1 (centurion)salary: R 30 000 R 35 000pm tctcExperience in the following industies: Vehicle tracking/fleet management/ saas/ big data or telematics highly advantageousbrief description of role:The main purpose of the
View Job
Data Science Engineer – Centurion
Alberton: Junior Data Analyst / Engineer (Jb4854) posted by Kontak Recruitment
"Junior Data Analyst / Engineer (JB4854) Alberton, Gauteng R10 000 to R15 000 a month CTCContract 6 to 12 months, which may become permanent A company committed to leveraging technology for data-driven insights and strategic intelligence solutions, offerin
View Job
Junior Data Analyst / Engineer (Jb4854)
Sandton: Data Engineer (Bi Analyst/ Etl Developer)
We are looking for Data Engineers to join our team based in JHB- onsite. The ideal candidate will have extensive experience in data engineering, particularly with SAS, SQL and Azure Cloud Stack, and will be responsible for designing, developing, and mainta
View Job
Data Engineer (Bi Analyst/ Etl Developer)
Pretoria: Junior Data Engineer – Centurion – R330k To R430k Per Annum
An AI and Fleet telematics that assist with your day-to-day operations, forecasting, reporting & compliance are in the market for a Junior Data Engineer. The ideal candidate must have a strong analytical background to help drive value from the companys dat
View Job
Junior Data Engineer – Centurion – R330k To R430k Per Annum
Cape Town City Centre: Mid Data Engineer
Responsibilities: Data Reporting: Design and maintain insightful reports to support key business decisions. Pipeline Development: Build and optimise scalable data pipelines for enhanced data accessibility. Collaboration: Work closely with cross-functional
View Job
Mid Data Engineer
Cape Town City Centre: Sap Data Engineer
The company helps organizations improve efficiency, drive digital transformation, and foster innovation across various industries. With a global presence, it is known for its customer-focused approach and expertise in using technology to address complex bu
View Job
Sap Data Engineer
Cape Town City Centre: Data Engineer
Key Responsibilities: Design, build, and maintain scalable data pipelines to support data integration, processing, and analytics. Collaborate with cross-functional teams to understand business requirements and implement data solutions. Work with large data
View Job
Data Engineer
Johannesburg: Data Engineer
As a Data Engineer, you will be responsible for designing, building, and maintaining data pipelines, databases, and infrastructure that enable high-quality data processing and analytics. You will work with cross-functional teams, including Data Scientists,
View Job
Data Engineer
Cape Town City Centre: Aws Data Engineer
PBT Group is currently offering an opportunity for an AWS Data Engineer with 4 years of relevant experience. The role of a Data Engineer involves constructing and maintaining data pipelines and datamarts, emphasizing scalability, repeatability, and securit
View Job
Aws Data Engineer
Johannesburg: Data Engineer
Duties and Responsibilities: Develop and maintain data pipelines Implement data integrations Perform data transformations Collaborate with cross-functional teams on data projects Ensure data quality and integrity Qualifications: Bachelor`s degree in Comput
View Job
Data Engineer
Centurion: Senior Data Engineer
Key Responsibilities Design & Build: Develop end-to-end data pipelines (ETL/ELT) to centralize and transform data from various sources. Data Mart Management: Design, implement, and optimize high-performing data marts for analytical and reporting needs. Clo
View Job
Senior Data Engineer
Johannesburg: Data Engineer – Johannesburg – R800k To R1mil Per Annum
Data Engineer Johannesburg - R800k to R1mil per annum Join a global financial services organisation whose core purpose is to seek out and invest in exceptional individuals who understand and support their core purpose, and whose values align and pride them
View Job
Data Engineer – Johannesburg – R800k To R1mil Per Annum
Midrand: Data Engineer
Position: Data Engineer Industry: Logistics Sector: Technology Location: Permanent Salary: Market related REQUIREMENTS: Matric or equivalent qualification A Bachelor`s degree or several years of experience in computer science, data science, software develo
View Job
Data Engineer
Cape Town City Centre: Data Engineer
Our client is currently seeking an experienced Data Engineer. The main focus of this role will be report writing and providing actionable insights. As the role evolves, the successful candidate will develop and maintain robust data pipelines to ensure seam
View Job
Data Engineer
Johannesburg: Data Engineer
TERTIARY QUALIFICATION(S) University Degree/College Diploma in the field of computer science or Informatics along with work experience in related field Essential Additional Computer Skills MicrosoftsAzure stack (ADF, Databricks,Azure SQLDatabase/Azure Syna
View Job
Data Engineer
Johannesburg: Data Engineer
My client from a Freight Forwarding Company is seeking a Data Engineer to join their team Responsibilities: The role is responsible for the design, development, testing, implementation, maintenance and support of the BI report and Data Warehouse solutions
View Job
Data Engineer
Johannesburg: Data Engineer
A reputable company in the Financial Services sector is seeking a Data Engineer to work with cloud data systems. Proficiency in the Microsoft BI Stack, T-SQL development, and cloud services is essential. Join a dynamic team and enhance your expertise while
View Job
Data Engineer
Kempton Park: Ref88265- Healthcare Azure Data Engineer (Road Logistics)
REF88265- Healthcare Azure Data Engineer (Road Logistics), Kempton Park, Rneg Ben Degree/Diploma Comp Science/ IT 5 years AZURE Development 8 years BI ETL Development 8 Years Advance SQL 5 years Data Warehouse Management MS Azure Stack BI Architcture Prin
View Job
Ref88265- Healthcare Azure Data Engineer (Road Logistics)
South Africa: Data Engineering (Us Working Hours)
The core advanced data engineering skillset is a comprehensive combination of technical expertise, platform knowledge, and problem-solving abilities required to build, maintain, and optimize robust, scalable, and efficient data systems. Data Architecture a
View Job
Data Engineering (Us Working Hours)
Cape Town City Centre: Senior Data And Vehicle Support Engineer (Telematics) Cape Town
Senior Data and Vehicle Support Engineer (Telematics) Cape Town Minimum requirements and experience: BEng or MEng (Electronic or Mechatronic Engineering) Minimum of 5 years working experience in telematics systems and -components including data managemen
View Job
Senior Data And Vehicle Support Engineer (Telematics) Cape Town
Centurion: Senior Data And Vehicle Support Engineer (Telematics) Centurion
Senior Data and Vehicle Support Engineer (Telematics) Centurion Minimum requirements and experience: BEng or MEng (Electronic or Mechatronic Engineering) Minimum of 5 years working experience in telematics systems and -components including data managemen
View Job
Senior Data And Vehicle Support Engineer (Telematics) Centurion
Johannesburg: Intermediate Azure Data Engineer – Johannesburg – R470k Up To R570k Per Annum
A consulting company is seeking an Intermediate Data Engineer to support a major client. In this role, you will execute data engineering tasks aligned with the clients established standards, frameworks, and roadmaps. If you have a solid background in data
View Job
Intermediate Azure Data Engineer – Johannesburg – R470k Up To R570k Per Annum
Johannesburg: Data Engineer (Fully Remote)
Data Architecture and Design - Data Modeling: o Create normalized and denormalized schemas (3NF, star, snowflake). o Design data lakes, warehouses, and marts optimized for analytical or transactional workloads. o Incorporate modern paradigms like data mesh
View Job
Data Engineer (Fully Remote)
Sandton: Data Engineer – Lumos – Sandton- 12 Months – Start Date – January 2025
Job Title: Data Engineer - Lumos - Sandton Location: Gauteng, Johannesburg Metro, Sandton Job Type: Contract Duration: 12 months Start Date: January 2025 Job Description Job Duties: Design and implement data solutions to support business requirements Devel
View Job
Data Engineer – Lumos – Sandton- 12 Months – Start Date – January 2025
Email Alerts
Click the button below and submit your email to receive email alerts with the latest jobs.
This service is offered by Result Generation
We do not save your information, and your email is the only information we forward to the service.
If you wish to unsubscribe, please click here
