Cloud Engineer

SUMMARY:
We are seeking a specialised Cloud Engineer to design, build, secure, and maintain the infrastructure powering our modern data platforms. Unlike a traditional Cloud Engineer focused on general application hosting, this role is dedicated to the unique demands of large-scale data workload, spanning high-volume storage, high-throughput networking, and compute-optimised environments for Spark and SQL-based processing.

POSITION INFO:

Responsibilities

Platform Architecture

• Design and implement scalable cloud infrastructure for data lakes and analytics platforms using Infrastructure as Code (Terraform)
• Define reusable, modular infrastructure patterns for repeatable deployments across environments

Security & Governance

• Implement identity and access management (IAM / Microsoft Entra ID) with least-privilege principles
• Design secure network architectures using VPC/VNet, Private Link, and endpoint-based access
• Enforce data governance standards using tools such as Unity Catalog

Automation & Delivery

• Build and maintain CI/CD pipelines for infrastructure and data platform components
• Automate provisioning of compute, workspaces, and platform services
• Enable policy-as-code and automated compliance checks within deployment workflows

Observability & Cost Management

• Implement monitoring and alerting for performance, reliability, and cost efficiency of data workloads
• Establish dashboards and controls for usage optimisation and capacity planning

Data Platform Enablement

• Create self-service infrastructure capabilities for Data Engineering teams
• Enable seamless deployment of pipelines without direct infrastructure dependency
• Support standardised patterns for scalable data product delivery

Requirements

• Minimum 5+ years of professional cloud engineering experience
• Strong exposure to data engineering environments and platforms (Databricks highly desirable)

Skills & Experience

Core Competencies

• Advanced expertise in Terraform (modules, state management, workspaces)
• Exposure to CloudFormation, Bicep, or Crossplane is beneficial

• Strong experience with Docker and Kubernetes (EKS, AKS, or equivalent)

• Advanced Git practices
• Experience with Azure DevOps and GitHub Actions for infrastructure automation and policy enforcement

• Strong proficiency in Python and Bash
• Experience with AWS CLI, Azure CLI, DBX CLI, and REST APIs

• Role-based access control (RBAC), least privilege design
• Encryption in transit and at rest
• Secrets management (Key Vault / Secrets Manager)

• Design and implementation of VPC/VNet architectures
• Private connectivity (Private Link/Endpoints), DNS, firewalls, and segmentation strategies

• Understanding of Medallion architecture, Delta Lake, and Data Vault concepts

• Cost monitoring, logging (CloudWatch / Azure Monitor), and alerting frameworks

• Familiarity with Spark, PySpark, SQL, dbt, and streaming workloads (e.g. Spark Structured Streaming)

Platform Expertise

AWS

• Storage: Advanced S3 configuration (lifecycle policies, access points, intelligent tiering)
• Identity: IAM role design, cross-account access, federation
• Networking: VPC architecture, Transit Gateways, Route53, Gateway Endpoints
• Serverless: Lambda and Step Functions automation
• Data Services: Glue, Kinesis, EMR

Azure

• Storage: ADLS Gen2 (ACLs, hierarchical namespace), Blob Storage
• Identity: Microsoft Entra ID, service principals, managed identities
• Networking: VNets, NSGs, Private Link architectures
• Data Services: Azure Data Factory, Synapse integration

Databricks Platform Engineering

• Workspace provisioning using Terraform provider
• Unity Catalog setup (metastores, catalogs, external locations)
• Cluster policies, instance profiles, and cost controls
• Secure connectivity (Private Link, VNet injection, IP access controls)
• Platform support for Jobs, Delta Live Tables (DLT), and orchestration tools
• Infrastructure enablement for MLflow, Mosaic AI, and Vector Search\

Certifications (Nice to Have)

• AWS Certified Solutions Architect – Associate / Professional
• Microsoft Certified: Azure Solutions Architect Expert
• HashiCorp Certified: Terraform Associate
• Certified Kubernetes Administrator (CKA)

• Databricks Certified Data Engineer Professional (highly desirable)
• AWS Certified Data Engineer – Associate
• Microsoft Azure Data Engineer Associate (DP-203)