Own and evolve the Databricks data platform, including Bronze, Silver, and Gold medallion layers, Delta Lake pipelines, source-system ingestion patterns, and enterprise data warehouse architecture.
Lead the transition toward event-driven, low-latency ingestion using Azure Event Hubs and Structured Streaming for source systems with demanding service-level requirements.
Establish and enforce standards for pipeline development, code quality, testing, documentation, and deployment.
Review architecture and code, mentor engineers, and guide sound technical decisions.
Partner with Data Architecture to incorporate lineage capture, quality checkpoints, and observability requirements into pipeline design.
Own the enterprise data warehouse as a production system, including data model integrity, pipeline reliability, ingestion service levels, and incident response.
Maintain and improve warehouse architecture, feed logic, and Gold-layer harmonization rules.
Ensure critical platform knowledge is documented and shared across the team.
Improve platform reliability, processing latency, scalability, and cost efficiency across all medallion layers.
Lead, develop, and retain Senior Data Engineers and Data Engineers, establishing clear ownership of critical platform capabilities.
Build a collaborative team through coaching, mentoring, individual development plans, and performance management.
Coordinate individual contributor hiring decisions with the Senior Director and define role profiles and hiring criteria.
Organize competing priorities and provide calm, clear leadership through production issues and changing demands.
Reinforce engineering discipline, documentation, accountability, and shared ownership of reliability.
Partner with Analytics Engineering on Gold-layer table design and Silver-to-Gold promotion logic supporting metric definitions and data products.
Work with QA to embed validation checkpoints and support Tier 1 and Tier 2 pipeline test suite development.
Collaborate with Data Governance on lineage, metadata, and data quality policy requirements.
Support Partner Data Services initiatives requiring new or modified pipelines.
Communicate engineering priorities, dependencies, and technical decisions effectively across peer functions.
Qualifications
Required
Seven or more years of progressive data engineering experience, including at least two years in people management or a team lead capacity. Qualified Team Leads and Tech Leads may be considered.
Deep, hands-on Databricks expertise, including medallion architecture, Delta Lake, Unity Catalog, and production pipeline development and operations.
Strong enterprise data warehouse expertise, including production design, operations, data modeling, and transformation logic.
Proficiency in PySpark, SQL, and Python sufficient to review code and architecture and mentor engineers.
Proven experience with Azure Databricks, Azure Data Lake Storage, and Azure Event Hubs.
Strong Kafka/Event Hubs experience and familiarity with Confluent Cloud.
Servant-leadership approach demonstrated through coaching, team development, and support for engineers.
Strong organization, prioritization, and communication skills, with the ability to manage multiple workstreams and remain composed under pressure.
Effective collaboration with functions such as QA, Data Governance, and Informatics.
Preferred
Healthcare, specialty pharmacy, or regulated-industry experience involving clinical or operational data.
Production experience with Structured Streaming and event-driven ingestion using Azure Event Hubs, Kafka, or equivalent technologies.
Familiarity with data contract standards such as ODCS and pipeline observability platforms such as Monte Carlo or Datadog