Description

Design and own the end-to-end architecture for a Databricks-based data lakehouse platform across Dev, UAT, and Prod environments.

This role is on-site.

Responsibilities

• Build batch and streaming data pipelines using Spark, Delta Lake, and Delta Live Tables, implementing medallion architecture patterns.
• Define and enforce platform best practices including CI/CD, version control, reusable libraries, and robust monitoring.
• Optimize Spark job performance through partitioning, caching, join strategies, and shuffle tuning to ensure reliability.
• Lead FinOps governance, managing cluster auto-scaling, cost allocation, budgets, and usage monitoring.
• Implement data governance, security controls (RBAC, row/column-level security), and lineage auditing via Unity Catalog.

Required Skills

• 10+ years of experience in Data Engineering and platform architecture.
• Deep expertise in Databricks, Spark, and Delta Lake for large-scale data processing.
• Hands-on experience with Kafka and structured streaming for real-time ingestion.
• Proficiency in cloud platforms (AWS, Azure, or GCP) and their data services.
• Strong SQL and RDBMS skills with a focus on data modeling.
• Proven track record in cost optimization and FinOps practices for cloud data platforms.
• Experience with Scala and PySpark for developing scalable data applications.

Preferred Skills

• Familiarity with AWS native services like Kinesis, SQS, and SNS.
• Knowledge of enabling AI/ML integrations, including RAG patterns and GenAI use cases.

Education: B.E/ B.Tech/ MCA

Key Skills

Education

Any Graduate