Description

You will build and maintain scalable ETL/ELT pipelines within an Azure environment.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from Azure Blob Storage, ADLS, and various relational or non-relational systems.
• Optimize Databricks workloads for performance and cost efficiency.
• Implement data security, access controls, and governance using Unity Catalog.
• Deploy Databricks jobs, notebooks, and configurations using Databricks Asset Bundles and GitLab.

Required Skills

• 5+ years of experience in data engineering roles.
• Expertise in Databricks, including Delta Lake, Unity Catalog, Lakehouse Architecture, and Delta Live Pipelines.
• Proficiency in PySpark and Spark for big data processing.
• Strong experience with Azure Cloud Services.
• Experience with relational databases.
• Hands-on use of Databricks Asset Bundles and GitLab for version control and deployment.
• Ability to collaborate with cross-functional teams to prepare data for BI tools and dashboards.

Preferred Skills

• Experience with Spark Streaming and real-time data solutions.
• Azure Data Engineer Associate or Databricks Certified Data Engineer Associate certification.

Key Skills

Education

Any Graduate