Description

You will manage and optimize end-to-end data workflows within the Azure ecosystem.

Responsibilities

• Manage and troubleshoot ADF and Databricks workflows, including triggers, linked services, and pipeline parameters.
• Debug Spark jobs and analyze notebook execution graphs to resolve complex failures.
• Optimize ADF pipeline performance, partitioning strategies, and ADLS data formats like Parquet.
• Automate data pipeline deployments using Azure DevOps, ARM templates, PowerShell, and Git.
• Govern data lifecycle rules and partition retention across raw and curated ADLS zones.
• Monitor resource consumption and implement cost-saving measures through auto-scaling and tiering.
• Prepare RCAs for P1/P2 incidents and coordinate UAT and rollback strategies.

Required Skills

• 5+ years of experience in data engineering roles.
• Expertise in Azure Data Factory and Databricks (PySpark).
• Proficiency in Python, PySpark, and SQL/SparkSQL.
• Hands-on experience with Azure Data Lake Storage (ADLS) and Azure Synapse.
• Strong knowledge of JSON configurations.
• Experience with Azure DevOps, Git, and CI/CD automation.
• Competency in Azure Monitor, diagnostics, and cost governance.
• Understanding of data security, IAM, RBAC, and audit enforcement.
• Any Graduate degree.

Preferred Skills

• Experience reviewing Power BI refresh bottlenecks and dataset tuning.

Key Skills

Education

Any Graduate