Description

You own the strategic design and architectural roadmap for the Enterprise Data Platform using Databricks and AWS, while remaining hands-on in design, coding, and performance tuning.

Responsibilities

• Lead architecture design and review sessions for strategic Databricks initiatives.
• Contribute code to critical components, building ETL/ELT pipelines using Python and PySpark.
• Prototype Proof of Concepts (PoCs) for new features like Databricks Asset Bundles or Delta Live Tables.
• Perform hands-on performance tuning and debugging of high-latency jobs using Spark UI and execution plans.
• Serve as the Technical Design Authority, making decisions on Serverless feature and AWS service adoption.

Required Skills

• 8-10 years of experience in Data Engineering/Architecture, with 4+ years specialized in Databricks and AWS.
• Expert knowledge of Data Lakehouse Architecture, Delta Lake internals, and Spark optimization.
• Advanced hands-on coding proficiency in Python and SQL; capable of production-ready code.
• Extensive experience designing solutions on the Databricks platform.
• Strong experience with Databricks Serverless architecture and Unity Catalog implementation.
• Deep understanding of Cloud Architecture (AWS Well-Architected Framework), including S3 and IAM.
• Proven experience designing complex ETL/ELT pipelines and Data Modeling (Dimensional, Vault, Relational).
• Experience in Design Review and Code Review processes, enforcing standards.
• Proficiency with PySpark, AWS, Python, SQL, and Scala.

Key Skills

Education

Bachelor's degree