Description

Project Overview
We are seeking an experienced Databricks Platform Engineering Lead to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.
The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.
The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.
Technical and Delivery Expertise
• 10+ years of overall technology experience.
• 5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.
• 3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
• Strong hands-on experience with:
o Azure Databricks
o Databricks account and workspace administration
o Unity Catalog
o Apache Spark and PySpark
o Python and SQL
o Delta Lake and Lakehouse architecture
o Azure Data Lake Storage
o Azure Data Factory or comparable integration tools
o Databricks Workflows and job orchestration
o Infrastructure as code using Terraform
o CI/CD and automated deployment practices
• Strong understanding of
o Databricks workspace provisioning and administration
o Development, test, and production environment management
o Compute provisioning and cluster configuration
o Cluster policies, pools, runtimes, and serverless compute
o Platform security, identity, and access management
o Service principals, managed identities, and secrets management
o Azure networking, private endpoints, and secure connectivity
o Unity Catalog architecture, permissions, lineage, and governance
o Platform monitoring, logging, auditability, and observability
o Databricks performance, capacity, and cost optimization
o Production support and platform-incident management
• Good understanding of data-engineering practices, including:
o Batch and streaming data ingestion
o ETL/ELT pipeline development
o Medallion architecture
o Data transformations and enrichment
o Data quality, validation, and reconciliation
o Schema management and data lineage
o Spark workload and query optimization
• Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.
• Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.
• Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance

Key Skills

Education

Any Graduate