GCP Data Engineer

SUMMARY:
We are looking for a skilled GCP Data Engineer to design, develop, and maintain scalable data pipelines and ETL processes on Google Cloud Platform (GCP). The successful candidate will be responsible for implementing and optimizing data storage solutions, deploying machine learning models, and maintaining data quality and consistency. You will work closely with cross-functional teams to drive data-driven solutions and improve operational efficiencies. This position is based in Johannesburg, South Africa, with a hybrid working model.

POSITION INFO:

Job Responsibilities:

• Design, develop, and maintain scalable data pipelines and ETL processes on Google Cloud Platform (GCP).
• Implement and optimize data storage solutions using BigQuery, Cloud Storage, and other GCP services.
• Collaborate with data scientists, machine learning engineers, data engineers, and other stakeholders to integrate and deploy machine learning models into production environments.
• Develop and maintain custom deployment solutions for machine learning models using tools such as Kubeflow, AI Platform, and Docker.
• Write clean, efficient, and maintainable code in Python and PySpark for data processing and transformation tasks.
• Ensure data quality, integrity, and consistency through data validation and monitoring processes.
• Deep understanding of Medallion architecture.
• Develop metadata-driven pipelines and ensure optimal processing of data.
• Use Terraform to manage and provision cloud infrastructure resources on GCP.
• Troubleshoot and resolve production issues related to data pipelines and machine learning models.
• Stay up-to-date with the latest industry trends and best practices in data engineering, machine learning, and cloud technologies, including data lifecycle management, data pruning, model drift, and model optimizations.

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Skills & Experience:

• Proven experience as a Data Engineer with a focus on Google Cloud Platform (GCP).
• Strong proficiency in Python and PySpark for data processing and transformation.
• Hands-on experience with machine learning model deployment and integration on GCP.
• Familiarity with GCP services such as BigQuery, Cloud Storage, Dataflow, and AI Platform.
• Experience with Terraform for infrastructure as code.
• Experience with containerization and orchestration tools like Docker and Kubernetes.
• Strong problem-solving skills with the ability to troubleshoot complex issues.
• Excellent communication and collaboration skills