Senior Data Engineer

Recruiter:

Network Recruitment

Job Ref:

ITE006391/Mon

Date posted:

Thursday, October 17, 2024

Location:

Johannesburg, South Africa

Salary:

Annually

SUMMARY:
Seeking a Senior Data Engineer who will be responsible for implementation and improvements.

POSITION INFO:
Responsibilities:

Identify and Implement Process Improvements: Automate manual tasks, optimize data delivery, and enhance technology scalability.
Enhance Data Systems: Implement new features and improvements across data platforms.
Streamline CI/CD Processes: Optimize data pipelines within continuous integration and delivery workflows.
Manage Complex Data Sets: Assemble large, complex datasets to meet business and technical requirements.
Optimize ETL Processes: Ensure efficient data extraction, transformation, and loading.
Conduct Testing for Data Consistency: Develop and execute unit tests to maintain data accuracy across pipelines.
Automate Monitoring: Build and maintain automated solutions for tracking data pipeline performance and health.
Support Analytics Infrastructure: Maintain and improve reporting and analytics systems.
Ensure Data Quality and Governance: Uphold data governance standards and maintain infrastructure (AWS, database security, etc.).
Manage Metadata and Documentation: Maintain the data warehouse, data lake metadata, catalogs, and user documentation.
Ensure Best Practices in Database Management: Maintain optimal configurations, including collations, engines, and indexing strategies.

Education:

5+ years' experience as a Data Engineer
BSc Degree in Computer Science / Information system / Engineering or related or equivalent experience
At least 2 years' experience building and optimising 'big data' data pipelines, architecture and maintaining data sets
Experience in Python, SQL, AWS, PySpark (desirable), AWS, Airflow, Docker
Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements

Apply now!

For more IT jobs, please visit />
If you have not had any response in two weeks, please consider the vacancy application unsuccessful. Your profile will be kept on our database for any other suitable roles / positions.

NB! This job is now closed. You can apply for other jobs by uploading your CV.

Similar jobs you might be interested in:

Senior Data Engineer
Location: Centurion
Salary: Market related
We are seeking a senior data engineer to join our team based in Centurion. Onsite, contract.
18 days ago

Senior Data Engineer
Location: Centurion
Salary:
senior data engineer – Lead a High-Impact data engineering Team! Are you a seasoned data engineering professional with a passion for leading technical teams? We are looking for a senior data engineer to take on a key leadership role in the development and delivery of large-scale data products and services. If you have hands-on experience in data engineering, Telemetry, or Fleet Management in...
62 days ago

Senior Specialist:Data Analytics
Location: Johannesburg
Salary: 1200000 Annually
senior Specialist: data Analytics Opportunity
5 days ago

Senior Data Warehouse Developer
Location: Centurion
Salary: Monthly
Psybergate is an IT company that builds bespoke software solutions and provides highly skilled resources to its clients.We are looking for a senior data Warehouse Developer to join our financial services client based in Centurion.
8 days ago

BI Developer
Location: Johannesburg
Salary: 500000
On the hunt for a Specialist: BI Developer (senior) in JHB!
2 days ago

Professional Civil Engineer - PrEng - ECSA Registered, Gauteng
Location: Johannesburg
Salary: 850000 Annually
Professional Civil engineer - PrEng - ECSA Registered, Gauteng
5 days ago

Senior Systems Developer
Location: Johannesburg
Salary: Annually
The senior Systems Developer will be responsible for designing, building, and maintaining scalable, secure and user-centric systems platforms to support consulting, reporting and the SaaS function.
7 days ago

Senior Back-End Developer
Location: Johannesburg
Salary: Annually
Seeking a Back-end C# Developer with strong experience in Azure cloud services and practical exposure to AI/LLM Integration. Building scalable APIs, Microservices and intelligent back-end features by leveraging Azure OpenAI, Cognitive service, vector search and modern .NET frameworks. Required: A strong engineering foundation with the ability to design AI-driven features that enhance automation, i...
8 days ago

Front-End Developer (Angular or React)
Location: Johannesburg
Salary: Annually
Seeking a Front-End engineer with strong React or Angular experience, who has hands-on experience integrating AI/LLM powered features into web applications. The ideal candidate combines excellent UI engineering skills with the ability to work with modern AI frameworks, APIs and prompt engineering techniques.
8 days ago

Release and DevOps Specialist
Location: Johannesburg
Salary: Annually
The Release and DevOps Specialist will be responsible for ensuring smooth, reliable and repeatable software releases from development, through to staging and production. It bridges the gap between development, QA and operations by managing release processes, automating deployments and maintaining a stable environment.
8 days ago