Description

You manage and optimize end-to-end data workflows within the Azure ecosystem.

This role is remote.

Responsibilities

• Manage and troubleshoot ADF and Databricks workflows, including triggers, linked services, and pipeline parameters.
• Debug Spark jobs and analyze notebook execution graphs to resolve complex failures.
• Optimize ADF pipeline performance, partitioning strategies, and ADLS data formats like Parquet.
• Automate data pipeline deployments using Azure DevOps, ARM templates, PowerShell, and Git.
• Monitor resource consumption, implement cost-saving measures, and prepare RCAs for P1/P2 incidents.

Required Skills

• 5+ years of experience in data engineering roles.
• Expertise in Azure Data Factory and Databricks (PySpark).
• Proficiency in Python, PySpark, and SQL/SparkSQL.
• Hands-on experience with Azure Data Lake Storage (ADLS) and Azure Synapse.
• Strong knowledge of JSON configurations and data security (IAM, RBAC).
• Experience with Azure DevOps, Git, and CI/CD automation.
• Competency in Azure Monitor, diagnostics, and cost governance.

Preferred Skills

• Experience reviewing Power BI refresh bottlenecks and dataset tuning.

Key Skills

Education

Any Graduate