Description

Key Responsibilities

• Design, build, and maintain batch and streaming data pipelines using Databricks (Spark, Delta Lake, Jobs/Workflows).
• Partner with data scientists, analysts, and application teams to deliver trusted, well-modeled data sets and features in the Databricks Lakehouse.
• Optimize Spark jobs (partitioning, caching, join strategies) and Databricks cluster configurations for performance, scalability, and cost.
• Implement data quality checks, observability, governance, and security controls (e.g., Unity Catalog, access policies) within Databricks.
• Troubleshoot and resolve pipeline failures, data issues, and production incidents; perform root-cause analysis and implement preventative improvements.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
• Proven experience delivering production data engineering solutions in Databricks, including Apache Spark and Delta Lake on the Microsoft Azure cloud platform.
• Strong proficiency in SQL and Python (or Scala), including building ETL/ELT pipelines and writing efficient Spark code.
• Experience with data modeling and lakehouse/warehouse patterns (e.g., medallion architecture) and distributed data processing concepts.
• Excellent problem-solving skills and attention to detail.
• Strong communication and teamwork abilities.

Preferred Skills

• Hands-on experience with Databricks ecosystem capabilities such as Unity Catalog, Delta Live Tables, MLflow, and/or Databricks SQL.
• Experience implementing CI/CD and automated testing for Databricks notebooks/repos and deployment of Jobs/Workflows.
• Knowledge of data security, privacy, and compliance controls in cloud lakehouse environments (including role-based access and auditing)

Key Skills

Education

Bachelor's degree