Senior Data Engineer (Spark & Python Specialist) - Remote

SUMMARY:
Our client is seeking a Senior Cloud Data Engineer to join their Engineering team at the company. In this role, you will be a key technical contributor responsible for building, optimizing, and maintaining high-performance data processing engines.
 
You will apply your deep expertise in Spark Best Practices to develop modular, portable PySpark applications, ensuring the companies platform remains provider-agnostic while operating within modern environments like Microsoft Fabric.
 
As an experienced member of the Data Engineering team, you will focus on refactoring legacy logic into scalable Python-centric solutions, optimizing the companies’ data lakehouse architecture, and contributing to the overall technical excellence of their data platform.

POSITION INFO:
Required Skills and Experience

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics & Dedicated SQL Pools: Solid experience working within Azure Synapse, specifically with Dedicate SQL Pools and Data Factory working with complex pipelines
• Data Lakehouse Expertise: Hands-on experience with Delta Lake and Parquet, specifically in high-volume environments.
• Agnostic Mindset: Experience working with open-source standards and containerization (Docker) to ensure data workloads can be ported across different cloud infrastructures.
• Collaborative Engineering: Proven track record of working within a team to successfully deliver and modernize large-scale data workloads.
• Ensure high levels of security, compliance, and data governance across data engineering processes.

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
• Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
• Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
• Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.

• Bachelor’s degree in computer science, Information Systems, Engineering, or a closely related field.

• Flexible Working Arrangements: Whether you’re a night owl or an early bird, the company offers hybrid and remote options to suit your lifestyle
• Comprehensive Benefits: From a wellness program to home office reimbursements and continuous learning opportunities, the company has got you covered.
• Team Culture: Fun team-building activities, regular socials, and a supportive, inclusive culture that values transparency, accountability, and work-life balance.
• Performance Incentives: Competitive salaries, ESOP, and recognition for your hard work.