Senior Data Engineer

SUMMARY:
-

POSITION INFO:

PBT Group has an opportunity for a Senior Data Engineer. You will be responsible for establishing new technology components and reusable solution patterns that can be leveraged by business facing development teams in their day-to-day solutions. You will constantly be developing and setting new principles, standards, processes, procedures, and guidelines for the wider BI community.

You should be able to communicate technical information to technical teams, as well as be competent in communicating challenges and solutions to project and operational leadership. An understanding of data management solutions and a keen sense of the strategic value of information to an organisation will be of importance. You will also be responsible for developing data warehousing blueprints, evaluating hardware and software platforms and integrating systems; translating business needs into long-term architecture solutions.

Duties:

• Define a structured approach to problem solving and delivery against it.
• Create role specific design standards, patterns and principles
• Assist the planning and management of the workload of the team and to ensure delivery
• Load large, complex data sets to and make data available for other data engineers
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing models for greater scalability
• Working with other data engineers and data modelers, you will design, implement, and manage data vaults, data transformations and the data pipeline
• Identify, design, and implement vault access layers to enable BI products to leverage the data within data vaults
• Monitor and fine-tune data vaults and data transformations on the Cloudera Hadoop stack
• Use modern development and modelling techniques and tools to implement BI and data management solutions, including data quality, metadata and reference data
• Engage with a wide range of technical stakeholders including data scientists, data analysts, business analysts, other data engineers and solutions architects
• Support data stewards to establish and enforce guidelines for data collection, quality improvements, integration, and processes

Experience:

• The Data Engineer must have at least 5 years’ experience in a similar environment working with the relevant tools and techniques
• The Data Engineer is someone with a strong understanding of data, data structures and data sources. Required skills include:
• Application and data engineering background with a solid background in data warehousing environments
• Experience in database management system (DBMS) physical implementation, including tables, joins and SQL querying.
• A solid background in Data Warehousing using SQL and ETL procedures is required, with a strong grounding in data modelling using either Kimball or Data Vault methodologies
• Experience in Hadoop Distributed File System, and experience in Spark, Oozie and NiFi are highly advantageous.
• Experience in data warehouse automation and the development of data pipelines using ETL tools (e.g. SAP Data Services), automation (e.g. Wherescape), scheduling and test automation (e.g. Robot) is desirable
• Experience of structured data, such as entities, classes, hierarchies, relationships and metadata.
• Experience in agile development
• Ability to comply to and manage data assets under a strict governance framework
• Strong Data Engineering background with a specific focus on staging high quality data
• Data Management technologies (e.g. Informatica Data Quality (IDQ), Informatica Enterprise Data Catalog (EDC), Axon, EBX) are preferred
• Data architecture design and delivery experience preferred

Qualifications/ Certification:

• Bachelor’s degree in Computer Science, Statistics, Informatics, Information Systems, Engineering or another quantitative field / National Diploma in an Information Technology related discipline preferred

NB! This job is now closed. You can apply for other jobs by uploading your CV.