Description

You will design, build, and maintain scalable data pipelines on Azure, integrating DevOps practices into the data engineering lifecycle.

Responsibilities

• Design and implement data pipelines using Azure Data Factory to orchestrate movement from databases and APIs into Azure Data Lake Storage and Blob Storage.
• Develop ETL processes within Azure Databricks clusters, utilizing Spark (PySpark/Scala) for data transformation.
• Build and maintain CI/CD pipelines for data infrastructure using Azure DevOps (Pipelines, Repos) and Bitbucket, ensuring automated testing and deployment.
• Provision and manage Azure infrastructure for data pipelines, including Databricks VMs and Azure Functions, using Infrastructure as Code (ARM/Bicep).
• Configure Azure Monitor and Azure Alerts to track pipeline health, performance, and security compliance.

Required Skills

• 2-5 years of experience in data engineering or analyst domains.
• Expert knowledge of Azure services: Data Factory, Databricks, Data Lake, Blob Storage, Azure Functions.
• Proficiency in Spark (PySpark/Scala) for data processing.
• Hands-on experience with Azure DevOps and Bitbucket for version control and CI/CD.
• Working knowledge of MS SQL and PostgreSQL.
• Experience with Change Data Capture (CDC) and data streaming applications like Apache Kafka and Qlik Replicate.
• Familiarity with Infrastructure as Code (ARM templates or Bicep).
• Understanding of Azure security best practices and compliance requirements.

Preferred Skills

• Bachelor’s degree in IT, Computer Science, or a relevant field.
• Ability to work independently in a complex, fast-paced environment.
• Strong verbal and written communication skills in English.

Key Skills

Education

Any Bachelors Degree from a recognized university