Description

You will serve as a subject matter expert for the Databricks platform, designing and implementing scalable ETL/ELT processes and complex data pipelines.

Responsibilities

• Design and implement data pipelines using Azure Data Factory or Qlik Replicate.
• Develop efficient ETL/ELT processes using Databricks notebooks with Python, Spark, or SQL.
• Build reliable data lake architectures using Delta Lake, implementing ACID transactions and schema enforcement.
• Optimize Spark jobs and cluster configurations for performance and cost efficiency.
• Enforce data governance and security policies using Unity Catalog.
• Maintain monitoring, alerting, and logging for Databricks jobs and clusters to proactively resolve pipeline issues.
• Champion code quality through version control, code reviews, and testing frameworks.

Required Skills

• 6+ years of experience in data engineering.
• Expert-level proficiency with Azure Databricks, including Notebooks, Jobs, and Workflows.
• Extensive experience with Spark (PySpark, Spark SQL) for large-scale data processing.
• Deep practical application of Delta Lake and Medallion architecture.
• Strong programming skills in Python and expert-level SQL proficiency.
• Experience with Azure Data Factory (ADF) and ADLS Gen2.
• Hands-on experience with Unity Catalog for data governance.
• Proficiency in data modeling (3NF, Dimensional, Kimball, or Inmon).
• Experience with Git/GitHub and CI/CD processes.
• Solid understanding of relational and NoSQL databases.

Preferred Skills

• Databricks Certifications (Associate or Professional).
• Experience with Scala, Talend, or Qlik Replicate.
• Knowledge of AWS services like Aurora, Postgres, Redshift, or S3.

Key Skills

Education

Bachelor's degree