Intermediate Data Engineer

SUMMARY:
Are you a data enthusiast with a passion for designing and optimizing robust data systems? We are looking for a skilled Data Engineer to join our team. Your expertise will drive the development and management of our cloud and on-premises data pipelines, ensuring the seamless flow of high-quality data across our organization.

POSITION INFO:
Intermediate Data Engineer Johannesburg Responsibilities: Develop and manage data pipelines to support data scientists, analysts, actuaries, and other initiatives. Ensure accurate, efficient, and cost-effective development of robust data systems in line with the organization's data strategy. Implement data validation processes to maintain clean, high-quality data. Establish and enforce guidelines for data collection, integration, and retention. Monitor data sources, processing, and environment for inconsistencies or anomalies, and resolve data quality problems. Leverage Azure DevOps Repos for version control and foster a collaborative development environment. Implement CI\/CD practices using Azure Pipelines to automate and streamline build and deployment processes. Ensure seamless integration between code repositories and deployment pipelines. Build and nurture relationships across the business to ensure alignment and commitment to data engineering objectives. Engage effectively with stakeholders to drive the success of data initiatives. Participate in assessment and development planning activities, formal and informal training, and coaching. Maintain an understanding of relevant technology, external regulation, and industry best practices through ongoing education and professional development. Up-skill in all facets of data engineering. Skills: Expert knowledge in Azure data services including Azure Data Factory, Azure Databricks, Data Lake, SQL Databases, Azure DevOps Boards and Repos. Proficiency in large-scale data processing with Apache Spark. Knowledge of GCP and AWS is desirable but not essential. Proficiency in Python and SQL is imperative. Exposure to PySpark is advantageous. Qualifications: Grade 12\/SAQA Accredited Equivalent (Essential) 3-year Degree in Information Technology\/Sciences\/Informatics\/Computer Sciences (Essential) Master's Degree in Information Technology\/Sciences\/Informatics\/Computer Sciences (Advantageous) Experience: 2 to 5 years of experience in data engineering, data science, or computer science (Essential) Experience in the Financial Services Industry, specifically Short-Term Insurance (Advantageous) We appreciate your interest, however if you have not had any response within 2 weeks, please consider your application unsuccessful. We also invite you to contact us to discuss your next career move in IT. IT Consultant

NB! This job is now closed. You can apply for other jobs by uploading your CV.