Description

You will design, build, and maintain batch and streaming data pipelines using Databricks, including Spark and Delta Lake.

This role is on-site.

Responsibilities

• Develop and optimize Spark jobs and Databricks cluster configurations for performance, scalability, and cost.
• Implement data quality checks, observability, governance, and security controls such as Unity Catalog and access policies.
• Partner with data scientists and analysts to deliver trusted, well-modeled datasets and features in the Databricks Lakehouse.
• Troubleshoot pipeline failures and production incidents, performing root-cause analysis to implement preventative improvements.

Required Skills

• 5+ years of experience delivering production data engineering solutions in Databricks on Microsoft Azure.
• Strong proficiency in SQL and Python for building efficient ETL/ELT pipelines and Spark code.
• Hands-on experience with Apache Spark and Delta Lake.
• Knowledge of data modeling, lakehouse/warehouse patterns (e.g., medallion architecture), and distributed data processing.
• Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
• Experience with CI/CD practices.

Preferred Skills

• Experience with Delta Live Tables, MLflow, or Databricks SQL.
• Knowledge of data security, privacy, and compliance controls in cloud lakehouse environments.

Key Skills

Education

Bachelor's degree