Description

Key Responsibilities
• Design enterprise-scale Lakehouse architecture using Databricks
• Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, and Databricks Workflows
• Implement Delta Lake, Medallion Architecture (Bronze/Silver/Gold), and Unity Catalog
• Lead migration of legacy data warehouses and ETL platforms to Databricks
• Design batch and real-time streaming data pipelines
• Optimize Spark jobs for performance, scalability, and cost efficiency
• Implement CI/CD pipelines and DevOps automation
• Define data governance, security, lineage, and access control standards
• Collaborate with business stakeholders, data engineers, analysts, and leadership teams
• Provide technical leadership, code reviews, architecture reviews, and best practices
• Support AI/ML enablement using MLflow and Databricks capabilities
• Establish monitoring, observability, and operational support frameworks

Required Skills
Technical Skills
• Strong hands-on experience with:
o Databricks
o Apache Spark
o PySpark
o SQL
o Python
• Experience with Delta Lake and Unity Catalog
• Expertise in data modeling and Lakehouse architecture
• Experience with cloud platforms:
o AWS
o Azure
o GCP
• Knowledge of:
o Data Lakes
o Data Warehousing
o Streaming frameworks
o CI/CD pipelines
o Git/DevOps
• Familiarity with orchestration tools:
o Airflow
o Azure Data Factory
o Databricks Workflows
Preferred Skills
• MLflow / AI & ML integration
• Kafka / Structured Streaming
• Snowflake integration
• Infrastructure as Code (Terraform)
• Kubernetes exposure
• Data governance and security frameworks

Experience Requirements
• 10+ years in Data Engineering / Data Architecture
• 5+ years working with Databricks
• Experience leading enterprise cloud modernization projects
• Strong stakeholder management and client-facing experience
• Experience designing scalable distributed data systems

Key Skills

Education

Any Gradute