Senior Data Engineering Technical Lead

SUMMARY:

POSITION INFO:
Key Responsibilities: Data Engineering & Migration Leadership: Design and deliver large-scale data migration strategies supporting identity transformation across multi-petabyte datasets. Develop data lineage mapping and validation frameworks to ensure migration accuracy, traceability, and data integrity. Build scalable batch and streaming data transformation pipelines with parallelized orchestration. Lead JDK and Apache Spark upgrade initiatives while maintaining production stability and performance. Implement monitoring, alerting, and observability capabilities to optimize data pipeline performance and system health. Create robust testing and validation frameworks to support complex migration and transformation scenarios. Technical Leadership & Engineering Excellence: Take technical ownership of critical platform and data engineering components. Mentor and support engineers through pair programming, code reviews, and collaborative problem-solving. Lead technical design and architecture discussions, ensuring alignment with engineering best practices and enterprise standards. Drive successful delivery by proactively identifying risks, dependencies, and blockers. Collaborate closely with cross-functional stakeholders to achieve technical alignment and project objectives. Required Experience & Qualifications: Minimum of 8 years' experience in Data Engineering or related software engineering roles. Bachelor's or Master's degree in Computer Science, Engineering, or equivalent practical experience. Demonstrated expertise in large-scale data migration programs and ETL pipeline development. Advanced proficiency in Scala and Java, with experience using Gradle or Maven. Strong hands-on experience with Apache Spark and distributed storage systems such as HDFS, Amazon S3, or Google Cloud Storage. Experience working with cloud-based data platforms including AWS, GCP, Azure, or Databricks. Practical experience with AWS EKS and Kubernetes-based deployments. Sound understanding of data privacy, security, and compliance regulations, preferably GDPR. Strong distributed systems knowledge, architecture expertise, and problem-solving capabilities. Preferred Experience: Experience working within large-scale enterprise engineering environments. Exposure to data lineage, metadata management, and dependency mapping frameworks. Experience with identity management and identity transformation systems. Knowledge of modern data formats including Iceberg, Parquet, ORC, and Avro. Experience with streaming technologies such as Kafka and Kinesis. Familiarity with observability and monitoring tools including Prometheus, Grafana, and Spark UI. Core Competencies: Software Design & Development Data Engineering & Data Migration Solution Architecture Testing & Quality Assurance Requirements Management Release & Deployment Management Monitoring & Observability Technical Leadership Mentorship & Coaching Cross-Functional Collaboration Culture & Team Fit: Successful candidates thrive in environments that value psychological safety, open communication, continuous learning, accountability, and collaborative problem-solving. The role requires a proactive mindset, constructive challenge, knowledge sharing, and a commitment to transparent teamwork. Why Join Us? Work on large-scale, cutting-edge data engineering and migration initiatives. Influence architecture and technical direction within a highly skilled engineering team. Access strong mentorship and leadership development opportunities. Collaborate with innovative organizations across technology, digital, and enterprise sectors. Enjoy a culture focused on continuous improvement, learning, and technical excellence. Benefit from flexible working arrangements and a healthy work-life balance. Competitive salary and benefits package with clear career progression opportunities. Apply now! For more IT jobs, please visit I also specialise in recruiting in the following: Software Development Data Engineer Data Analyst Infrastructure Architecture ... and more! If you have not had any response in two weeks, please consider the vacancy application unsuccessful. Your profile will be kept on our database for any other suitable roles \/ positions. For more information contact: Sfundo Shoba Recruitment Consultant \/>