Data Engineering Lead(Big Data)

SUMMARY:

The purpose of this role is to lead the technical delivery team, using experience to deliver right sized and well architected solutions in order to meet targets. The successful candidate will be expected to use their experience working in Azure to understand the available technology choices and selectively implement services safely and securely....

POSITION INFO:

The purpose of this role is to lead the technical delivery team, using experience to deliver right sized and well architected solutions in order to meet targets. The successful candidate will be expected to use their experience working in Azure to understand the available technology choices and selectively implement services safely and securely. As well as demonstrable experience working with the technologies and processes, such as Data Bricks, Python, Unity catalog, DevOps (CI/CD) and Plotly.

The company’s Edge platform has been built, from the ground up, to cater for advanced analytics and AI solutions. As such, the successful candidate will be able to work with leading data Scientists, within the company’s Data Services team or those that work for our strategic partners and clients.

Internal Process

• Responsible for the end-to-end technical solution. This means that key decisions, once discussed and debated, are appropriately understood, and implemented.
• Collaborate with the Product Development Lead and the Business Enablement Lead to ensure that all stakeholder expectations are met.
• Ensure that the company’s Edge platform can cater to all varieties of data, including batch and real-time, and is safe, secure, scalable, and stable.
• Manage the work items on behalf of the team, breaking down and assigning work items per sprint. Managing elements of sprint planning.
• DevOps CI/CD pipeline management, control, and oversight.
• Engage other platform owners and agree on optimal integration paths and methods.
• Outstanding communication and presentation skills, with the ability to convey complex concepts to non-technical stakeholders.

Client Services

• Ongoing relationship building with internal and external stakeholders to ensure smooth business processing.
• Ability to develop new improved processes and to facilitate discussions with external and internal stakeholders.
• Committed to Client experience.
• Enable business communications between business areas.
• Participate and contribute to a culture that builds rewarding relationships, facilitates feedback, and provides exceptional client service.
• Build trust and creditability with respect to Data in the financial market.

People Management

• Develop and maintain productive and collaborative working relationships with colleagues, peers, and stakeholders.
• Continuously develop own expertise in terms of professional, industry, and legislation knowledge.
• Contribute to continuous innovation through the development, sharing, and implementation of new ideas.
• Ability to work proactively and independently to deliver precise and accurate work.
• Take ownership of driving personal career development.
• Comfortable working with a strictly agile project methodology. This includes breaking down features and user stories for the team of data engineers into appropriately sized deliverables and when required, defining standards, performing code reviews, and validating for quality.
• Display an inquisitive, detailed-orientated nature, and be unafraid to ask difficult questions, or challenge the status quo when the need arises.
• Responsible for leveraging financial industry experience to build a modern, secure, scalable solution that is appropriate for the clients and partners that the company services.
• Makes use of, and develops cost-effective, right-sized solutions and services within Microsoft Azure’s Cloud Platform to meet the core platform needs.
• Displays consideration of the increasing risks faced as a society and as an organisation and suggests appropriate responses with regard to the solutions being built.

 

Finance

• Identify solutions to enhance cost -effectiveness and increase operational efficiency.
• Provide input into the risk identification processes and communicate recommendations in the appropriate forum.

• Grade 12 or equivalent
• A bachelor’s degree in computer science or another closely related field
• 5-8 years of working experience in a related field.
• Proven work experience as a senior data engineer, or as a Data Engineering Lead
• Strong Data Bricks and Python skills are a must. SQL skills as an alternative to Python will also be considered
• Understand how big data solutions / NoSQL solutions differ from standard data solutions and explain why choices made might be different.
• Mature experience working with cloud services: Azure, AWS, or GCP. This experience will allow the candidate to propose appropriate solutions and alternatives.
• Strong DevOps understanding, including CI/CD