Data Engineer

Recruiter:

Recru-it

Job Ref:

PE010826/LM

Date posted:

Friday, June 7, 2024

Location:

Johannesburg, South Africa

Salary:

market related Monthly

SUMMARY:
Job Purpose:
The purpose of the Data Engineer is to leverage their data expertise and data related technologies, in line with the company Data Architecture Roadmap, to advance technical thought leadership for the Enterprise, deliver fit for purpose data products, and support data initiatives. In addition, Data Engineers enhance the data infrastructure of the company to enable advanced analytics, machine learning and artificial intelligence by providing clean, usable data to stakeholders. They also create data pipelines, Ingestion, provisioning, streaming, self-service, API and solutions around big data that support the company strategy to become a data driven organisation.

Job Responsibilities:

Responsible for the maintenance, improvement, cleaning, and manipulation of data in the companies operational and analytics databases.
Data Infrastructure: Build and manage scalable, optimised, supported, tested, secure, and reliable data infrastructure e.g. using Infrastructure and Databases (DB2, PostgreSQL, MSSQL, HBase, NoSQL, etc.), Data Lakes Storage (Azure Data Lake Gen 2), Cloud-based solutions (SAS, Azure Databricks, Azure Data Factory, HDInsight), Data Platforms (SAS, Ab Initio, Denodo, Netezza, Azure Cloud). Ensure data security and privacy in collaboration with Information Security, CISO and Data Governance
Data Pipeline Build (Ingestion, Provisioning, Streaming and API): Build and maintain data pipelines to:
create data pipelines for data integration (Data Ingestion, Data Provisioning and Data Streaming) utilising both On Premise tool sets and Cloud Data Engineering tool sets
efficiently extract data (Data Acquisition) from Golden Sources, Trusted sources and Writebacks with data integration from multiple sources, formats and structures
load the companies Data Warehouse (Data Reservoir, Atomic Data Warehouse, Enterprise Data Mart)
provide data to the respective Lines of Business Marts, Regulatory Marts and Compliance Marts through self-service data virtualisation
provide data to applications or companies Data consumers
transform data to a common data model for reporting and data analysis, and to provide data in a consistent, useable format to companies data stakeholders
handle big data technologies (Hadoop), streaming (KAFKA) and data Replication (IBM Inphosphere Data Replication)
drive utilisation of data integration tools (Ab Initio) and Cloud data integration tools (Azure Data Factory and Azure Data Bricks)
Data Modelling and Schema Build: In collaboration with Data Modellers, create data models and database schemas on the Data Reservoir, Data Lake, Atomic Data Warehouse and Enterprise Data Marts.
The Companies Data Warehouse Automation: Automate, monitor and improve the performance of data pipelines.
Collaboration: Collaborate with Data Analysts, Software Engineers, Data Modelers, Data Scientistsm Scrum Masers and Data Warehouse teams as part of a squad to contribute to the data architecture detail designs and take ownership of Epics end-to-end and ensure that data solutions deliver business value.
Data Quality and Data Governance: Ensure that reasonable data quality checks are implemented in the data pipelines to maintain a high level of data accuracy, consistency and security.
Performance and Optimisation: Ensure the performance of the companies data warehouse, integration patterns, batch and real time jobs, streaming and API's.
API Development: Build API's that enable the Data Driven Organisation, ensuring that the data warehouse is optimised for API's by collaborating with Software Engineers.

Essential Qualifications - NQF Level:

Advanced Diplomas/National 1st Degrees

Preferred Qualification:

Field of Study: BCom, BSc, BEng

Preferred Certifications:

Cloud (Azure, AWS), DEVOPS or Data engineering certification. Any Data Science certification will be an added advantage, Coursera, Udemy, SAS Data Scientist certification, Microsoft Data Scientist.

Minimum Experience Level:

Total number of years of experience:3 - 6 years
Type of experience: Experienced at working independently within a squad and has the demonstrated knowledge and skills to deliver data outcomes without supervision.
Experience designing, building, and maintaining data warehouses and data lakes.
Experience with big data technologies such as Hadoop, Spark, and Hive.
Experience with programming languages such as Python, Java, and SQL.
Experience with relational databases and NoSQL databases.
Experience with cloud computing platforms such as AWS, Azure, and GCP. Experience with data visualization tools. Result-driven, analytical creative thinker, with demonstrated ability for innovative problem solving.

Technical / Professional Knowledge

Cloud Data Engineering (Azure, AWS, Google)
Data Warehousing
Databases (PostgreSQL, MS SQL, IBM DB2, HBase, MongoDB)
Programming (Python, Java, SQL)
Data Analysis and Data Modelling
Data Pipelines and ETL tools (Ab Initio, ADB, ADF, SAS ETL)
Agile Delivery
Problem solving skills

Behavioral Competencies

Decision Making
Influencing
Communication
Innovation
Technical/Professional Knowledge and Skills
Building Partnerships
Continuous Learning

POSITION INFO:
Job Purpose:
The purpose of the Data Engineer is to leverage their data expertise and data related technologies, in line with the company Data Architecture Roadmap, to advance technical thought leadership for the Enterprise, deliver fit for purpose data products, and support data initiatives. In addition, Data Engineers enhance the data infrastructure of the company to enable advanced analytics, machine learning and artificial intelligence by providing clean, usable data to stakeholders. They also create data pipelines, Ingestion, provisioning, streaming, self-service, API and solutions around big data that support the company strategy to become a data driven organisation.

Job Responsibilities:

Responsible for the maintenance, improvement, cleaning, and manipulation of data in the companies operational and analytics databases.
Data Infrastructure: Build and manage scalable, optimised, supported, tested, secure, and reliable data infrastructure e.g. using Infrastructure and Databases (DB2, PostgreSQL, MSSQL, HBase, NoSQL, etc.), Data Lakes Storage (Azure Data Lake Gen 2), Cloud-based solutions (SAS, Azure Databricks, Azure Data Factory, HDInsight), Data Platforms (SAS, Ab Initio, Denodo, Netezza, Azure Cloud). Ensure data security and privacy in collaboration with Information Security, CISO and Data Governance
Data Pipeline Build (Ingestion, Provisioning, Streaming and API): Build and maintain data pipelines to:
create data pipelines for data integration (Data Ingestion, Data Provisioning and Data Streaming) utilising both On Premise tool sets and Cloud Data Engineering tool sets
efficiently extract data (Data Acquisition) from Golden Sources, Trusted sources and Writebacks with data integration from multiple sources, formats and structures
load the companies Data Warehouse (Data Reservoir, Atomic Data Warehouse, Enterprise Data Mart)
provide data to the respective Lines of Business Marts, Regulatory Marts and Compliance Marts through self-service data virtualisation
provide data to applications or companies Data consumers
transform data to a common data model for reporting and data analysis, and to provide data in a consistent, useable format to companies data stakeholders
handle big data technologies (Hadoop), streaming (KAFKA) and data Replication (IBM Inphosphere Data Replication)
drive utilisation of data integration tools (Ab Initio) and Cloud data integration tools (Azure Data Factory and Azure Data Bricks)
Data Modelling and Schema Build: In collaboration with Data Modellers, create data models and database schemas on the Data Reservoir, Data Lake, Atomic Data Warehouse and Enterprise Data Marts.
The Companies Data Warehouse Automation: Automate, monitor and improve the performance of data pipelines.
Collaboration: Collaborate with Data Analysts, Software Engineers, Data Modelers, Data Scientistsm Scrum Masers and Data Warehouse teams as part of a squad to contribute to the data architecture detail designs and take ownership of Epics end-to-end and ensure that data solutions deliver business value.
Data Quality and Data Governance: Ensure that reasonable data quality checks are implemented in the data pipelines to maintain a high level of data accuracy, consistency and security.
Performance and Optimisation: Ensure the performance of the companies data warehouse, integration patterns, batch and real time jobs, streaming and API's.
API Development: Build API's that enable the Data Driven Organisation, ensuring that the data warehouse is optimised for API's by collaborating with Software Engineers.

Essential Qualifications - NQF Level:

Advanced Diplomas/National 1st Degrees

Preferred Qualification:

Field of Study: BCom, BSc, BEng

Preferred Certifications:

Cloud (Azure, AWS), DEVOPS or Data engineering certification. Any Data Science certification will be an added advantage, Coursera, Udemy, SAS Data Scientist certification, Microsoft Data Scientist.

Minimum Experience Level:

Total number of years of experience:3 - 6 years
Type of experience: Experienced at working independently within a squad and has the demonstrated knowledge and skills to deliver data outcomes without supervision.
Experience designing, building, and maintaining data warehouses and data lakes.
Experience with big data technologies such as Hadoop, Spark, and Hive.
Experience with programming languages such as Python, Java, and SQL.
Experience with relational databases and NoSQL databases.
Experience with cloud computing platforms such as AWS, Azure, and GCP. Experience with data visualization tools. Result-driven, analytical creative thinker, with demonstrated ability for innovative problem solving.

Technical / Professional Knowledge

Cloud Data Engineering (Azure, AWS, Google)
Data Warehousing
Databases (PostgreSQL, MS SQL, IBM DB2, HBase, MongoDB)
Programming (Python, Java, SQL)
Data Analysis and Data Modelling
Data Pipelines and ETL tools (Ab Initio, ADB, ADF, SAS ETL)
Agile Delivery
Problem solving skills

Behavioral Competencies

Decision Making
Influencing
Communication
Innovation
Technical/Professional Knowledge and Skills
Building Partnerships
Continuous Learning

NB! This job is now closed. You can apply for other jobs by uploading your CV.

Similar jobs you might be interested in:

Data Engineer
Location: Pretoria
Salary: 850 000 Annually
Are you a skilled data engineer ready to build scalable, high-performance data solutions? Join a dynamic team where your expertise in modern data platforms, cloud technologies, and big data processing will directly support business intelligence, analytics, and client reporting. This is your opportunity to work on impactful projects and grow your career in a forward-thinking environment.
1 day ago

Data Engineer
Location: Centurion
Salary:
Are you a cloud-native data engineer who thrives on building scalable, high-performance data solutions? This is your chance to join a forward-thinking organisation that’s reshaping how data drives business decisions.
6 days ago

Data Engineer
Location: Johannesburg
Salary: Negotiable
The QA Analyst/engineer will play a vital role in delivering trusted solutions that support business decision-making.
7 days ago

Kafka & Flink Data Engineer
Location: Johannesburg
Salary:
7 days ago

Senior Data Engineer
Location: Centurion
Salary: 700 000 Annually
Are you a seasoned data engineering professional ready to lead high-impact projects in a cutting-edge telemetry and fleet management environment? This is your opportunity to take ownership of technical delivery, drive innovation, and mentor top talent in a role that blends leadership with hands-on engineering excellence.
10 days ago

Data Engineer
Location: Sandton
Salary: Market Related
data engineer
13 days ago

Senior Data Engineer
Location: Centurion
Salary:
Senior data engineer – Lead a High-Impact data engineering Team! Are you a seasoned data engineering professional with a passion for leading technical teams? We are looking for a Senior data engineer to take on a key leadership role in the development and delivery of large-scale data products and services. If you have hands-on experience in data engineering, Telemetry, or Fleet Management in...
15 days ago

Senior Data Engineer
Location: Centurion
Salary:
Are you a seasoned data engineer and visionary professional ready to lead high-impact projects in a dynamic, data-driven environment? We’re looking for a technically strong Senior data engineer to join our client and drive innovation across their data platforms.
15 days ago

Data Engineer
Location: Pretoria
Salary:
Introduction:Our client, a fast-growing data solutions start-up based in Pretoria East, is looking for a motivated and detail-oriented data engineer to join their dynamic team. This is an exciting opportunity for someone passionate about data, technology, and innovation to contribute to building scalable data systems and play a key role in shaping the company’s data engineering capability.&n...
16 days ago

Data Engineer/Analyst
Location: Pretoria
Salary:
We’re recruiting for a prominent and highly respected bank, offering a 24-month contract opportunity for an experienced data engineer/Analyst. This role is part of the bank’s transformational programme to enhance process efficiency, effectiveness, and quality through business process modelling and engineering.We're looking for a sharp analytical thinker with strong interpersonal skills...
17 days ago