Description

You will lead the design, implementation, and optimization of large-scale data pipelines within an Azure environment.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from Azure Blob Storage, ADLS, and various relational or non-relational systems.
• Optimize Databricks workloads for performance and cost efficiency.
• Manage data security, access controls, and governance using Unity Catalog.
• Deploy Databricks jobs, notebooks, and configurations using Databricks Asset Bundles.

Required Skills

• 5+ years of experience in data engineering.
• Expertise in Databricks, including Delta Lake, Unity Catalog, Lakehouse Architecture, and Delta Live Pipelines.
• Proficiency in Python, PySpark, and Spark for big data processing.
• Strong SQL skills and experience with relational databases.
• Hands-on experience with Azure Cloud Services and ETL processes.
• Experience with Databricks Asset Bundles and GitLab for version control.
• Knowledge of MDM, Snowflake, and ETL frameworks.
• Degree in any field.

Preferred Skills

• Familiarity with Spark Streaming and real-time data solutions.
• Azure Data Engineer Associate or Databricks Certified Data Engineer Associate certification.

Key Skills

Education

Any Gradute