Senior Data Engineer (Spark & Python Specialist)

SUMMARY:
Role:
Our client is seeking a Senior Cloud Data Engineer to join their Engineering team at the company. In this role, you will be a key technical contributor responsible for building, optimizing, and maintaining high-performance data processing engines.
 
You will apply your deep expertise in Spark Best Practices to develop modular, portable PySpark applications, ensuring the companies platform remains provider-agnostic while operating within modern environments like Microsoft Fabric.
 
As an experienced member of the Data Engineering team, you will focus on refactoring legacy logic into scalable Python-centric solutions, optimizing the companies’ data lakehouse architecture, and contributing to the overall technical excellence of their data platform.
 
Key Responsibilities

• Spark Implementation & Optimization: Act as a senior technical resource for Spark internals, applying best practices in memory management, shuffle tuning, and partitioning to ensure their Spark-based processing is performant and cost-effective.
• Cloud-Agnostic Development: Develop and maintain data pipelines using Python, PySpark, and Delta Lake/Parquet, adhering to their strategy of decoupling code from specific cloud provider services and reducing reliance on GUI-based tools like ADF.
• Refactoring & Modernization: Contribute to the ongoing evolution of the companies platform by refactoring complex SQL-based ETL into modular, testable, and maintainable Python libraries.
• Lakehouse Engineering: Build and optimize Medallion Architecture (Bronze/Silver/Gold) layers using Delta Lake, ensuring efficient data versioning, schema evolution, and storage performance (Z-Ordering, Vacuuming).
• Code-First Orchestration: Support the transition toward code-centric orchestration patterns (e.g., Airflow, Dagster, or Python-based wrappers) that prioritize portability and reduce dependency on cloud-specific orchestration tools.
• Technical Excellence: Participate in code reviews, mentor junior engineers in PySpark best practices, and contribute to the development of automated testing frameworks (Pytest) to ensure high-quality, reliable data delivery.

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics & Dedicated SQL Pools: Solid experience working within Azure Synapse, specifically with Dedicate SQL Pools and Data Factory working with complex pipelines
• Data Lakehouse Expertise: Hands-on experience with Delta Lake and Parquet, specifically in high-volume environments.
• Agnostic Mindset: Experience working with open-source standards and containerization (Docker) to ensure data workloads can be ported across different cloud infrastructures.
• Collaborative Engineering: Proven track record of working within a team to successfully deliver and modernize large-scale data workloads.
• Ensure high levels of security, compliance, and data governance across data engineering processes.

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
• Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
• Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
• Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.

• Bachelor’s degree in computer science, Information Systems, Engineering, or a closely related field.

• Flexible Working Arrangements: Whether you’re a night owl or an early bird, the company offers hybrid and remote options to suit your lifestyle
• Comprehensive Benefits: From a wellness program to home office reimbursements and continuous learning opportunities, the company has got you covered.
• Team Culture: Fun team-building activities, regular socials, and a supportive, inclusive culture that values transparency, accountability, and work-life balance.
• Performance Incentives: Competitive salaries, ESOP, and recognition for your hard work.

POSITION INFO:
Role:
Our client is seeking a Senior Cloud Data Engineer to join their Engineering team at the company. In this role, you will be a key technical contributor responsible for building, optimizing, and maintaining high-performance data processing engines.

You will apply your deep expertise in Spark Best Practices to develop modular, portable PySpark applications, ensuring the companies platform remains provider-agnostic while operating within modern environments like Microsoft Fabric.

As an experienced member of the Data Engineering team, you will focus on refactoring legacy logic into scalable Python-centric solutions, optimizing the companies’ data lakehouse architecture, and contributing to the overall technical excellence of their data platform.

Key Responsibilities

• Spark Implementation & Optimization: Act as a senior technical resource for Spark internals, applying best practices in memory management, shuffle tuning, and partitioning to ensure their Spark-based processing is performant and cost-effective.
• Cloud-Agnostic Development: Develop and maintain data pipelines using Python, PySpark, and Delta Lake/Parquet, adhering to their strategy of decoupling code from specific cloud provider services and reducing reliance on GUI-based tools like ADF.
• Refactoring & Modernization: Contribute to the ongoing evolution of the companies platform by refactoring complex SQL-based ETL into modular, testable, and maintainable Python libraries.
• Lakehouse Engineering: Build and optimize Medallion Architecture (Bronze/Silver/Gold) layers using Delta Lake, ensuring efficient data versioning, schema evolution, and storage performance (Z-Ordering, Vacuuming).
• Code-First Orchestration: Support the transition toward code-centric orchestration patterns (e.g., Airflow, Dagster, or Python-based wrappers) that prioritize portability and reduce dependency on cloud-specific orchestration tools.
• Technical Excellence: Participate in code reviews, mentor junior engineers in PySpark best practices, and contribute to the development of automated testing frameworks (Pytest) to ensure high-quality, reliable data delivery.

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics & Dedicated SQL Pools: Solid experience working within Azure Synapse, specifically with Dedicate SQL Pools and Data Factory working with complex pipelines
• Data Lakehouse Expertise: Hands-on experience with Delta Lake and Parquet, specifically in high-volume environments.
• Agnostic Mindset: Experience working with open-source standards and containerization (Docker) to ensure data workloads can be ported across different cloud infrastructures.
• Collaborative Engineering: Proven track record of working within a team to successfully deliver and modernize large-scale data workloads.
• Ensure high levels of security, compliance, and data governance across data engineering processes.

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
• Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
• Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
• Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.

• Bachelor’s degree in computer science, Information Systems, Engineering, or a closely related field.

• Flexible Working Arrangements: Whether you’re a night owl or an early bird, the company offers hybrid and remote options to suit your lifestyle
• Comprehensive Benefits: From a wellness program to home office reimbursements and continuous learning opportunities, the company has got you covered.
• Team Culture: Fun team-building activities, regular socials, and a supportive, inclusive culture that values transparency, accountability, and work-life balance.
• Performance Incentives: Competitive salaries, ESOP, and recognition for your hard work.