Intermediate Data Engineer

SUMMARY:
Are you a data enthusiast with a passion for designing and optimizing robust data systems? We are looking for a skilled Data Engineer to join our team. Your expertise will drive the development and management of our cloud and on-premises data pipelines, ensuring the seamless flow of high-quality data across our organization.

POSITION INFO:
Intermediate Data Engineer
Johannesburg

Responsibilities:

• Develop and manage data pipelines to support data scientists, analysts, actuaries, and other initiatives.
• Ensure accurate, efficient, and cost-effective development of robust data systems in line with the organization's data strategy.
• Implement data validation processes to maintain clean, high-quality data.
• Establish and enforce guidelines for data collection, integration, and retention.
• Monitor data sources, processing, and environment for inconsistencies or anomalies, and resolve data quality problems.
• Leverage Azure DevOps Repos for version control and foster a collaborative development environment.
• Implement CI/CD practices using Azure Pipelines to automate and streamline build and deployment processes.
• Ensure seamless integration between code repositories and deployment pipelines.
• Build and nurture relationships across the business to ensure alignment and commitment to data engineering objectives.
• Engage effectively with stakeholders to drive the success of data initiatives.
• Participate in assessment and development planning activities, formal and informal training, and coaching.
• Maintain an understanding of relevant technology, external regulation, and industry best practices through ongoing education and professional development.
• Up-skill in all facets of data engineering.

• Expert knowledge in Azure data services including Azure Data Factory, Azure Databricks, Data Lake, SQL Databases, Azure DevOps Boards and Repos.
• Proficiency in large-scale data processing with Apache Spark.
• Knowledge of GCP and AWS is desirable but not essential.
• Proficiency in Python and SQL is imperative.
• Exposure to PySpark is advantageous.

• Grade 12/SAQA Accredited Equivalent (Essential)
• 3-year Degree in Information Technology/Sciences/Informatics/Computer Sciences (Essential)
• Master’s Degree in Information Technology/Sciences/Informatics/Computer Sciences (Advantageous)

• 2 to 5 years of experience in data engineering, data science, or computer science (Essential)
• Experience in the Financial Services Industry, specifically Short-Term Insurance (Advantageous)

NB! This job is now closed. You can apply for other jobs by uploading your CV.