Description

You will design and lead complex big data architecture initiatives with a focus on cost optimization and platform migration.

Responsibilities

• Conduct cost analysis and audit existing Databricks deployments to implement strategies for 30-50% savings.
• Design and deploy automated monitoring solutions and cost allocation frameworks for business unit accountability.
• Lead strategic migrations from Databricks to alternatives like Spark on Kubernetes, EMR, or GCP DataProc.
• Develop high-performance, cost-optimized Spark cluster architectures outside of the Databricks ecosystem.
• Collaborate with finance teams to establish multi-year cost reduction roadmaps and business cases with ROI calculations.

Required Skills

• 8+ years in big data architecture with a focus on cost optimization.
• 5+ years of hands-on Databricks experience with proven cost reduction achievements.
• 6+ years of advanced Apache Spark development and cluster management experience.
• Track record of achieving significant cost savings (minimum 40%+) in cloud data platforms.
• Expert knowledge of Databricks pricing models and cost drivers.
• Deep expertise in migrating complex data workloads between different Spark platforms.
• Experience with Kubernetes-based Spark deployments and container orchestration.
• Proficiency in Scala, Python, and Java.
• Familiarity with AWS, GCP, Prometheus, and Grafana.

Key Skills

Education

Bachelor's or Master's degrees