Description

Architect and manage Databricks environments, data pipelines, and platform operations.

This role is hybrid.

Responsibilities

• Architect and manage Databricks environments, Unity Catalog, and Delta Lake.
• Design and optimize data pipelines, workflows, and automation frameworks.
• Monitor DBU usage, forecasting, and cost optimization strategies.
• Support Unix/Linux administration, shell scripting, and platform operations.
• Implement governance, monitoring, HA/DR, and compliance best practices.

Required Skills

• 7+ years in Data Engineering/Data Platforms with 4+ years of Databricks experience.
• Expertise in Databricks, PySpark, Python, Delta Lake, DLT, and SQL Warehouses.
• Strong Unix/Linux administration and Bash scripting experience.
• Experience with AWS/Azure/GCP cloud platforms.
• Knowledge of MLflow, AutoML, AI/ML concepts, and automation frameworks.
• Experience with Oracle, Git, Jira, ServiceNow, and ETL tools like Fivetran/Ab Initio.

Preferred Skills

• Experience with Generative AI/LLM workflows.
• Terraform/Ansible and enterprise orchestration tools.
• Insurance or financial domain experience.

Key Skills

Education

Bachelor's degree