Description

You will design, build, and manage scalable data processing pipelines within the Azure ecosystem.

Responsibilities

• Design and implement scalable data pipelines using API and streaming ingestion methods.
• Manage and optimize data storage across Azure Data Lake, Azure SQL Data Warehouse, and Azure Cosmos DB.
• Automate workflows to streamline data ingestion, processing, and distribution.
• Monitor and troubleshoot data issues to maintain high availability and performance.
• Implement data security measures including encryption, access controls, and auditing.
• Document data architectures, procedures, and systems to ensure compliance.

Required Skills

• 10+ years of experience in data engineering.
• Expert level SQL for writing and optimizing complex queries.
• Deep expertise in Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service.
• Proven experience with Big Data technologies including Spark, Kafka, and Hive.
• Hands-on experience developing NoSQL solutions using Azure Cosmos DB.
• Strong understanding of ETL design and complex data modelling.
• Experience building pipelines for low-latency stream and dataset processing.
• Knowledge of DevOps processes, CI/CD, and Infrastructure as Code.
• Thorough understanding of Azure and AWS Cloud Infrastructure.

Preferred Skills

• Working knowledge of Python.

Key Skills

Education

Any Graduate