Data Engineer III

SUMMARY:
Data Engineer III

POSITION INFO:

Roles / Responsibilities:

• Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment
• Design and develop programmatic transformations of the solution, by correctly partitioning,  formatting and validating the data quality
• Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the datalake
• Provide operational support to datamart datafeeds and datamarts
• Design infrastructure required to develop and operate datalake data feeds
• Design infrastructure required to develop and operate datamarts, their user interfaces and the feeds required to populate the datalake.

JOB RELATED EXPERIENCE

• Business Intelligence (5+ years)
• Extract Transform and Load (ETL) processes  (5+ years)
• Cloud AWS (2+ years)
• Agile exposure, Kanban or Scrum (2+ years)
• Planning and team leading skills
• Strong focus on data engineering, i.e. building pipelines, big data environments

JOB RELATED KNOWLEDGE

• Creating data feeds from on-premise to AWS Cloud (2 years)
• Support data feeds in production on break fix basis (2 years)
• Creating data marts using Talend or similar ETL development tool (4 years)
• Manipulating data using python and pyspark (2 years)
• Processing data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop (2 years)
• Devop for Big Data and Business Intelligence including automated testing and deployment (2 years)

JOB RELATED SKILLS

• Talend
• AWS: EMR, EC2, S3
• Python
• PySpark or Spark
• BI Data Modelling
• SQL

NB! This job is now closed. You can apply for other jobs by uploading your CV.