Description

You will build and manage scalable data pipelines and cloud ETL processes within an Azure environment.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks.
• Transform and process large datasets using PySpark, Spark, and SQL.
• Integrate data from Azure Blob Storage, ADLS, and various relational or non-relational systems.
• Optimize Databricks workloads for cost efficiency and performance.
• Implement data security, access controls, and governance using Unity Catalog.
• Deploy Databricks jobs, notebooks, and configurations using Databricks Asset Bundles.

Required Skills

• 5+ years of experience in data engineering.
• Expertise in Databricks, including Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, and Delta Live Pipelines.
• Proficiency in PySpark, Spark, and SQL.
• Strong experience with Python and ETL/ELT development.
• Hands-on experience with Azure Cloud Services and Datawarehouse design.
• Experience managing version control for Databricks artifacts.
• Ability to prepare data for BI tools and dashboards.

Key Skills

Education

Any Gradute