Description

You will architect and build scalable Lakehouse solutions, owning the end-to-end lifecycle of distributed data pipelines.

This role is remote.

Responsibilities

• Build and optimize distributed Spark pipelines using PySpark, focusing on partitioning, caching, broadcast joins, execution plan tuning, and memory optimization.
• Implement resilient batch and streaming pipelines with robust ETL/ELT patterns, including retry logic, observability, logging, and alerting.
• Design and manage change data capture (CDC) and incremental load mechanisms to ensure data freshness and integrity.
• Establish CI/CD practices for data workflows, utilizing Infrastructure as Code (IaC) and Git-based automated deployments.
• Own the performance and reliability of data infrastructure, ensuring scalability and maintainability of the analytics platform.

Required Skills

• 5+ years of professional experience in data engineering, with a strong focus on distributed systems.
• Deep expertise in Apache Spark and PySpark, including advanced tuning techniques for performance and memory management.
• Proven experience building scalable batch and streaming data pipelines with complex ETL/ELT logic.
• Hands-on experience with CI/CD tools and Infrastructure as Code (IaC) for automating data workflow deployments.
• Proficiency with Git for version control and collaborative development.
• Strong understanding of data lakehouse architectures and best practices.

Key Skills

Education

Any Graduate