Description

Senior Databricks Data Engineer owning distributed data processing, partitioning, and parallelism strategies for client solutions.

Responsibilities

• Design and implement data systems tailored to specific client requirements and business processes.
• Architect and implement CI/CD pipelines for Databricks notebooks and jobs, including testing and versioning.
• Define and enforce coding standards for ETL processes to ensure maintainability and reusability.
• Develop disaster recovery strategies for Databricks environments to ensure data resilience.
• Lead global delivery teams using Scrum and Agile methodologies to manage backlogs and ceremonies.

Required Skills

• 8+ years of experience in architecture, design, and implementation using Databricks.
• Expertise in Databricks and AWS environments.
• Deep knowledge of Spark internals, Catalyst optimization, and the Databricks runtime environment.
• Proficiency in SQL and Python.
• Hands-on experience with Delta Lake storage and Spark (PySpark, Spark SQL).
• Proven ability to design scalable, fault-tolerant systems.
• Experience deploying Databricks on cloud platforms with advanced configurations.
• Strong background in integrating data across disparate systems and platforms.

Preferred Skills

• Experience with Airflow, Splunk, Kubernetes, Power BI, Git, or Azure DevOps.
• Background in the Insurance (P&C) industry.

Key Skills

Education

Any Graduate