Data Engineer

SUMMARY:
Psybergate is an IT company that builds bespoke software solutions and provides highly skilled resources to its clients.
We are looking for a Data Engineer to join our financial services client based in Sandton for a 12-months contract role.

POSITION INFO:
What you will be doing:
We are looking for an experienced Data Engineer with strong Python and SQL skills, as well as hands-on experience with Kafka, Docker, and Kubernetes. The ideal candidate will have a proven track record in building bespoke data processing pipelines, developing microservices, and designing data table structures and schemas. You will collaborate with cross-functional teams to integrate and optimize data systems, while working with both structured and unstructured data.

Key Responsibilities:

• Design and Build Data Pipelines: Develop and maintain custom, scalable data pipelines for processing large datasets, including unstructured text and document data.
• Integrate Data Systems: Collaborate with diverse teams to understand data sources and build robust integrations with various internal and external systems.
• Build Microservices: Develop microservices to handle data processing tasks, ensuring they are scalable, secure, and efficient.
• Optimize Performance: Design and optimize table structures, schemas, and database queries for improved data retrieval and storage performance.
• Kafka and Real-time Data: Leverage Kafka for real-time data streaming and processing, ensuring high availability and low-latency data delivery.
• Containerization & Orchestration: Use Docker and Kubernetes for deploying and managing data applications in production environments.
• Collaborate with Teams: Work closely with data scientists, analysts, and other engineers to define requirements and deliver high-quality, reliable data solutions.
• Ensure Data Quality: Maintain data integrity and implement best practices to ensure the consistency and quality of the data being processed

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
• 5 years experience as Date Engineer
• Strong proficiency in Python: Experience in developing scalable data applications, automating processes, and implementing algorithms.
• Strong SQL skills: Ability to design efficient queries and optimize database performance.
• Hands-on experience with Kafka, including setting up and managing streams, and integrating with data pipelines.
• Proficiency with Docker and Kubernetes for containerization and orchestration in production environments.
• Experience building bespoke data processing pipelines: Expertise in handling large volumes of data from multiple sources and optimizing processing workflows.
• Experience building microservices to deliver modular, scalable, and maintainable data solutions.
• Experience designing table structures and schemas that meet both performance and business requirements.
• Ability to collaborate with diverse teams, understanding their needs and building integrations that allow for efficient data flow.
• Java experience is advantageous, particularly for implementing additional data processing logic or integration points.
• Experience with unstructured text and document data is a plus, especially in processing and transforming such data for analytical purposes.

NB! This job is now closed. You can apply for other jobs by uploading your CV.