Description

You will bridge the gap between data engineering and infrastructure automation within the Microsoft Azure ecosystem.

Responsibilities

• Design, build, and maintain large-scale data processing pipelines using Azure Data Factory, Data Lake, Synapse, and Databricks.
• Lead the implementation of Azure DevOps CI/CD pipelines for automated build, test, and deployment processes.
• Implement Infrastructure as Code (IaC) using Terraform or ARM templates to automate cloud infrastructure management.
• Manage data ingestion, transformation, and orchestration using ETL/ELT processes and Azure-native services.
• Troubleshoot and optimize data workflows using Azure Monitor, Log Analytics, and Application Insights.

Required Skills

• 10-13 years of experience in data engineering, cloud architecture, and DevOps.
• Expertise in Azure DevOps, including CI/CD pipeline development and Git version control.
• Hands-on experience with Azure Data Lake, Azure SQL, Azure Synapse Analytics, Azure Databricks, and Azure Data Factory.
• Proficiency in Python, PySpark/SparkSQL, Java or Scala, and SQL.
• Experience with big data technologies such as Spark, Hive, and Kafka.
• Strong experience with Terraform, ARM templates, and PowerShell scripting for automation.
• Knowledge of data security and compliance standards including GDPR and HIPAA.
• Experience managing data migrations and orchestrations in large-scale environments.
• Familiarity with SQL Server, AzureSQL, or MySQL.

Preferred Skills

• Certifications in Azure DevOps or Azure Data Engineer.
• Familiarity with AWS or Google Cloud platforms.

Key Skills

Education

Not specified