Cape Town Region: Databricks Data Engineer posted by Intelligent Debt Management

Position : Data Engineer – Intermediate (Hybrid/Pinelands, Cape Town) The Intelligent Debt Management (IDM) Group, South Africas largest and multi-award winning debt management company, has a position available for aData Engineer. This individual will join the Data Team and will be accountable for data engineering design, data ingestion and support across on-premise and cloud-based platforms. Whats in it for you? Being South Africas largest company of its kind IDM is continuously growing, with no signs of stopping. At IDM, we build on current-generation platforms like Databricks and Microsoft Fabric, giving you hands-on exposure to tools and patterns that are shaping the future of data engineering. This is not a legacy maintenance role youll be involved in designing, building, and evolving data solutions from the ground up, with real room for creativity and engineering judgement. Youll work alongside a team of high caliber senior data engineers, where knowledge sharing is part of how we operate. Technical discussions are encouraged, opinions are respected, and decisions are driven by merit rather than hierarchy. Our team is actively involved in AI and machine learning initiatives, giving you exposure beyond traditional data pipelines. We value clean architecture, thoughtful design, and creativity. You will have the opportunity to influence how things are built, contribute to architectural decisions, and help shape the evolution of our data platform. Culturally, we are a diverse, respectful, and collaborative team. We invest in people, support continuous learning, and create an environment where engineers can grow both technically and professionally. Whether you want to deepen your engineering expertise or broaden your impact across the business, youll find room to do so here. Beyond the work itself, IDM offers a stable, growing environment where the data you build helps real South Africans improve their financial outcomes giving purpose and impact to the systems you design. Other key benefits: We have a vibrant office in a lively part of the city, meaning you have easy access to stores and outlets if you need. We offer company benefits medical aid, retirement annuity, Bursaries and Insurance We have a culture of growth, meaning you have access to move both upwards or across in the business if you want to. Requirements Candidates must be able to demonstrate: 3 years experience as a data engineer or in a similar role , designing and implementing data pipelines, data integration and ETL processes. Practical experience in Azure, Azure Data Factory and Azure Databricks is preferred. Proven experience inPython/PySparkandSQL/SparkSQLprogramming skills. At least 3 years of experience in Information Technology specifically in technical data roles e.g. data engineering, data analytics or data science. Knowledge of data management and storage solutions, including SQL Server Database, Azure Data Lake Storage or similar. Having good communication skills and being able to operate in a team environment. Excellent attention to detail and problem-solving skills related to data, data structures and algorithms. Key responsibilities will include: Contribute to the architecture and design of the IDM Data Platform and ingestion patterns. Implement and support the scalable data solutions on the Microsoft Azure cloud platform, including Azure Databricks. Develop, maintain and optimise robust production data pipelines for data ingestion, processing, and transformation to support analytics, machine learning and other downstream data-driven services. Ensure data security, quality and integrity across all of the pipelines and related data processes. Collaborate with the data product owner, analysts, data scientists, and business stakeholders to understand data requirements and implement effective data workflows. Assist in data initiatives and support technical teams including architects, analysts, software developers, and data scientists. Qualifications : Bachelor’s degree in Computer Science, Engineering, or a related field. Certifications in Azure, Microsoft Fabric Data Engineering, Azure Databricks Data Engineering or similar or equivalent practical experience.