Description

You will lead the design and implementation of scalable and secure data processing pipelines.

Responsibilities

• Design and implement scalable and secure data processing pipelines using Azure Data Factory, Azure Databricks, and other Azure services.
• Manage and optimize data storage using Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB.
• Monitor and troubleshoot data-related issues within the Azure environment to maintain high availability and performance.
• Automate data pipelines and workflows to streamline data ingestion, processing, and distribution tasks.
• Utilize Azure's analytics services, such as Azure Synapse Analytics, to provide insights and support data-driven decision-making.

Required Skills

• 10+ years of professional experience in data engineering.
• Expert level understanding of Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service.
• Expert level skills in writing and optimizing complex SQL.
• Fluent with Big Data technologies such as Apache Spark and Apache Kafka.
• Proficiency in Python for data processing applications.
• Experience designing and building ETL/ELT solutions.
• Knowledge of Dev-Ops processes, including CI/CD practices.
• Experience with complex data modeling and large database environments.

Key Skills

Education

Any Graduate