Data Engineer

SUMMARY:
We are sourcing for a Data Engineer role.

POSITION INFO:
Duties and Responsibilities: Contribute to the establishment and operation of the AI Centre of Excellence (AI CoE) Collaborate with executives to develop strategies that align with organisational imperatives. Partner with AI Champions, ICT, Data Scientists and AI\/ML Engineers to integrate AI into operational workflows smoothly. Data Ingestion & Integration Build scalable batch and streaming pipelines from internal and external sources (APIs, CDC, files, events). Implement schema evolution, late-arriving data handling, and idempotent ingestion. Set up connectors and jobs across cloud services, message queues (Kafka\/Kinesis\/Event Hubs), and integration tools. Data Transformation & Modeling Design and maintain curated datasets (bronze\/silver\/gold) using ELT\/ETL. Implement dimensional models, data vault, and\/or lakehouse patterns as appropriate. Use transformation frameworks (e.g., dbt, Spark) with robust testing and documentation. Data Platform Engineering Build and operate data lakes \/ warehouses \/ lake houses (e.g., Delta Lake, BigQuery, Synapse). Manage storage formats and optimizations (Parquet\/ORC, partitioning, compaction, etc). Implement cataloguing, lineage, and discovery (e.g., Unity Catalog, Purview, Data Catalogue). Reliability, Observability & Quality Implement data testing (unit, expectations), monitoring (freshness, volume, schema, distribution), and alerting (e.g., Great Expectations, Soda, Monte Carlo). Establish SLAs\/SLOs and incident response for data pipelines. Ensure reproducibility, versioning, and backfills with orchestration (e.g., Airflow, Dagster, Prefect). Security, Governance & Compliance Apply data access controls (RBAC\/ABAC), masking, tokenization, encryption at rest\/in transit. Enforce data retention, lineage, and regulatory compliance (POPIA or industry-specific). Collaborate with Data Governance and Security teams on policies and audits. Cost & Performance Optimization (FinOps) Monitor compute\/storage spend and optimize partitioning, caching, spot\/preemptible usage. Right-size warehouses\/clusters and tune jobs for performance and cost efficiency. Collaboration & Enablement Partner with Data Scientists\/Analysts on feature engineering, model-ready tables, and MLOps handoffs. Create self-service patterns, templates, and documentation to accelerate downstream teams. Contribute to platform standards, reusable components, and developer experience. Provide mentorship to emerging data engineers Education, Skills and Experience A diploma or bachelor's degree in computer science, Information Systems, Information Technology, Engineering, or a related field (or equivalent experience) is required. 6+ years in Data Engineering or similar roles (Software Engineer with data focus, ETL Developer). Experience with Lakehouse platforms (Databricks, Snowflake, BigQuery, Synapse\/OneLake). Hands-on with Terraform\/Bicep\/CloudFormation for infrastructure-as-code. Familiarity with MLOps (feature stores, model registries, batch\/stream scoring integration). Proven experience building and operating production-grade data pipelines