Data Engineer - Developer (Python) - Cape Town

Recruiter:

5th Dimension Consulting

Job Ref:

Data Engineer - Deve..

Date posted:

Tuesday, January 5, 2021

Location:

CapeTown, South Africa

Salary:

negotiable

SUMMARY:
Seeking a Data Engineer - Developer (Python) with an IT Degree/Diploma plus AWS Certification at least to associate level, plus 5+ years experience in Business Intelligence & ETL processes, for a position with a large retail company in Cape Town.

POSITION INFO:

Job Purpose

Build and support data pipelines and datamarts built off those pipelines. Both must be scalable, repeatable and secure.
Help to facilitate gathering data from a variety of different sources, in the correct format, assuring that it conforms to data quality standards and assuring that downstream users can get to that data timeously. This role functions as a core member of an agile team.
Responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly.
Enable solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces.
Additional responsibilities include developing proof of concepts and implements complex big data solutions with a focus on collecting, parsing, managing, analysing and visualising large datasets.

Responsibilities

Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment
Design and develop programmatic transformations of the solution, by correctly partitioning, formatting and validating the data quality
Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the datalake
Provide operational support to datamart datafeeds and datamarts
Design infrastructure required to develop and operate datalake data feeds
Design infrastructure required to develop and operate datamarts, their user interfaces and the feeds required to populate the datalake.

Requirements

Qualifications

IT Degree/Diploma
AWS Certification at least to associate level

Experience

Experience in the Retail environment is advantageous
5+ years experience in Business Intelligence & ETL processes
2+ years experience in Agile (eKanban or Scrum)
2+ years Cloud AWS experience

Essential Knowledge (at least 2 years)

Creating data feeds from on-premise to AWS Cloud.
Support data feeds in production on break fix basis.
Creating data marts using Talend or similar ETL development tool.
Manipulating data using python and pyspark.
Processing data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop.
Devop for Big Data and Business Intelligence including automated testing and deployment.

Skills

Talend
AWS: EMR, EC2, S3
Python
Business Intelligence Data modelling
SQL
PySpark or Spark (desirable)

NB! This job is now closed. You can apply for other jobs by uploading your CV.

Similar jobs you might be interested in:

Senior Data Engineer
Location: Stellenbosch
Salary: Annually
Blast off into data galaxies by designing and building pipelines that fuel smart business decisions.Work with GCP technologies to craft scalable architectures across Africa’s fastest-growing fintech.Be the astronaut of analytics—turning raw data into constellations of insight.
1 day ago

Data Engineer (GCP)
Location: Cape Town
Salary:
3 days ago

Senior Cloud Data Engineer
Location: Cape Town
Salary:
What will you do?You will be working primarily within the Google Cloud Environment using a variety of the tools that Google offers from Biquery and dataproc to Kubernettes and AI Hub. You should idea dream in python and speak SQL. You should not be afraid to dive into dirty data and help the team make sense of it. We are in the game of taking data and turning it into amazing stories and pretty pic...
4 days ago

Intermediate Cloud Data Platform Support Engineer
Location: Cape Town
Salary:
7 days ago

Data Engineer
Location: Somerset West
Salary:
data engineer vacancy in Belville, cape town.Must have a Bachelor’s degree in an engineering, Programming, or data Science field and at least 3 years of relevant work experience in data engineering, Development or related fields. Experience working with GCP (Google Cloud Platform), SQL and Google BigQuery.
8 days ago

Data Scientist / Senior Software Engineer
Location: Stellenbosch
Salary:
Are you excited to tackle groundbreaking projects blending data science and software engineering? My client, a global pioneer in cutting-edge technology, seeks a Senior Software engineer with data Science expertise to architect and implement AI and ML features for large-scale, real-time systems. If you thrive on transforming vast datasets into actionable insights, this is your opportunity to make ...
8 days ago

Data Engineer
Location: Stellenbosch
Salary: 1 000 000 Annually
SOON... the world will kneel.In the bleak corridors of cyberspace, beneath the blinking lights of powerless analysts, a new empire is rising. And my client — the Architect of Algorithms, the Lord of Latency, the Baron of Bandwidth — is building an army.But they require a Senior data engineer.Not just any code monkey…They seek a conduit of chaos. A manipulator of models. A warden...
8 days ago

Data Engineer
Location: Stellenbosch
Salary: Annually
Seeking a data engineer with strong Google Cloud Platform (GCP) experience. The data engineer will be responsible for designing, building and maintaining scalable data pipelines and architecture that powers the business.
11 days ago

Data Engineer
Location: Stellenbosch
Salary:
We are seeking an experienced data engineer with strong expertise in Google Cloud Platform to join a fast-growing, innovative organisation. This role offers the chance to design, build, and optimise scalable data pipelines and architectures that support impactful decision-making across the business.
14 days ago

Data Engineer
Location: Stellenbosch
Salary: TBC Annually
An innovative financial services company redefining credit in Africa is searching for a talented data engineer with strong Google Cloud Platform (GCP) expertise. With operations across Zambia, Kenya, Uganda, and South Africa, this is your chance to build scalable data solutions that empower smarter decisions and real impact across the continent.
14 days ago

Data Engineer - Developer (Python) - Cape Town

Similar jobs you might be interested in:

Create a free job alert for Data Engineer - Developer (Python) - Cape Town in CapeTown