Description

You will design and implement scalable data pipelines and ETL processes on the Databricks platform.

This role is on-site.

Responsibilities

• Design and implement data pipelines using Azure Data Factory or Qlik Replicate.
• Develop efficient ETL/ELT processes using Databricks notebooks with Python, Spark, or SQL.
• Build reliable data lake architectures using Delta Lake, implementing ACID transactions and schema enforcement.
• Optimize Spark jobs and cluster configurations for performance and cost efficiency.
• Enforce data governance and security policies using Unity Catalog and maintain monitoring for pipeline issues.

Required Skills

• 6+ years of experience in data engineering.
• Expert-level proficiency with Azure Databricks, including Notebooks, Jobs, and Workflows.
• Extensive experience with Spark (PySpark, Spark SQL) for large-scale data processing.
• Deep practical application of Delta Lake and Medallion architecture.
• Strong programming skills in Python and expert-level SQL proficiency.
• Experience with Azure Data Factory (ADF) and ADLS Gen2.
• Hands-on experience with Unity Catalog for data governance.
• Proficiency in data modeling (3NF, Dimensional, Kimball, or Inmon).
• Experience with Git/GitHub and CI/CD processes.

Preferred Skills

• Databricks Certifications (Associate or Professional).
• Experience with Scala, Talend, or Qlik Replicate.
• Knowledge of AWS services like Aurora, Postgres, Redshift, or S3.

Key Skills

Education

Bachelor's degree