Senior Redshift Data Engineer

SUMMARY:
-

POSITION INFO:

Job Summary: PBT Group is seeking a Senior Redshift Data Engineer with expertise in AWS Redshift to join our team. The Senior Redshift Data Engineer will play a crucial role in driving business impact by designing, developing, and optimizing data pipelines and ETL processes for our AWS Redshift-based data lake house. This individual will collaborate closely with cross-functional teams, employ best practices, and ensure the seamless ingestion, integration, transformation, and orchestration of data.

Key Responsibilities:

• Collaborate with data engineering and development teams to design, develop, test, and maintain robust and scalable ELT/ETL pipelines using SQL scripts, Redshift stored procedures, and other AWS tools and services.
• Work closely with engineering and data teams to understand business requirements and data integration needs, translating them into effective data solutions.
• Architect, implement, and manage end-to-end data pipelines, ensuring data accuracy, reliability, data quality, performance, and timeliness.
• Utilize AWS DMS and other services for efficient data ingestion from on-premises databases into Redshift.
• Design and implement ETL processes, including Changed Data Capture (CDC) and Slow Changing Dimension (SCD) logics, to seamlessly integrate data from diverse source systems.
• Provide expertise in Redshift database optimization, performance tuning, and query optimization.
• Design and implement efficient orchestration workflows using Airflow for complex ETL processes.
• Integrate Redshift with other AWS services, such as AWS DMS, AWS Glue, AWS Lambda, Amazon S3, Airflow, and more, to build end-to-end data pipelines.
• Perform data profiling and analysis to troubleshoot data-related challenges/issues and build solutions.
• Identify opportunities to automate tasks and develop reusable frameworks.
• Maintain a well-organized and documented repository of codes, scripts, and configurations using Git.
• Provide technical guidance and mentorship to fellow developers, sharing best practices and optimization techniques for Redshift-based data solutions.

Qualifications and Skills:

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• Minimum of 5 years of hands-on experience in designing, developing, and maintaining data pipelines and ETL processes on AWS Redshift, including data lakes and data warehouses.
• Proficiency in SQL programming and Redshift stored procedures for efficient data manipulation and transformation.
• Hands-on experience with AWS services such as AWS DMS, Amazon S3, AWS Glue, Redshift, Airflow, and other pertinent data technologies.
• Strong understanding of ETL best practices, data integration, data modeling, and data transformation.
• Experience with complex ETL scenarios, including CDC and SCD logics, and integrating data from multiple source systems.
• Expertise in AWS DMS for seamless ingestion from on-prem databases to AWS cloud.
• Proficiency in Python programming with a focus on developing efficient Airflow DAGs and operators.
• Experience in converting Oracle scripts and Stored Procedures to Redshift equivalents.
• Familiarity with version control systems, particularly Git, for maintaining a structured code repository.
• Strong problem-solving skills, attention to detail, and a focus on data quality and accuracy.
• Ability to collaborate effectively in a fast-paced, agile environment and communicate technical concepts to non-technical stakeholders.
• Proven track record of delivering high-quality data solutions within designated timelines.
• Experience working with large-scale, high-volume data environments.
• AWS certifications related to data engineering or databases are a plus.

Preferred Qualifications and Skills:

• Working experience with both relational and NoSQL databases.
• Working experience with data virtualization technologies like Starburst.
• Advanced understanding of database backup, recovery, and archiving strategies.
• Advanced knowledge of linear algebra, statistics, and geometrical algorithms.
• Strong communication and collaboration skills, including the ability to work effectively with cross-functional teams.

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

NB! This job is now closed. You can apply for other jobs by uploading your CV.