Description

You will build and manage data integration pipelines and processing logic within the Azure ecosystem.

Responsibilities

• Implement Change Data Capture (CDC) mechanisms using Python and PySpark when native Azure Data Factory support is unavailable.
• Design and configure Azure Data Factory pipelines using Lookup, Foreach, If, Switch, Metadata, and Web activities.
• Manage Data Flow operations and set up Self-Hosted Integration Runtimes to handle concurrent pipeline executions.
• Optimize SQL performance through indexing, partitioning, and complex queries using analytical functions.
• Develop custom logic using Azure Functions and manage Databricks clusters to prevent memory and timeout issues.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency in Python and PySpark for data processing.
• Hands-on experience with Azure Data Factory (ADF) and Data Flows.
• Strong advanced SQL skills, including stored procedures and performance tuning.
• Experience with Databricks cluster management and memory optimization.
• Knowledge of Azure Synapse and Azure Functions.
• Ability to troubleshoot cluster memory issues and implement custom loading strategies for unsupported databases.

Preferred Skills

• Degree in any graduate field.

Key Skills

Education

Any Gradute