Description

Lead data engineering initiatives focused on Azure Databricks and large-scale data orchestration.

Responsibilities

• Design and implement data pipelines using Azure Databricks for cleaning, transformation, and loading into Azure Synapse Analytics.
• Orchestrate data ingestion from various sources into Azure Synapse and Azure Data Lake Storage (ADLS) using Azure Data Factory (ADF).
• Execute end-to-end implementation of data warehouses and data marts.
• Write SQL queries to analyze Type-2 dimension data.
• Apply Medallion Architecture patterns using Azure services.

Required Skills

• 4+ years of experience in Information Technology.
• 3+ years of hands-on experience with Azure Data Factory (ADF) for data ingestion into Azure Synapse and ADLS.
• 2+ years of experience with Azure, Databricks, and Hadoop distributed frameworks.
• 2+ years of experience designing data pipelines using Azure Databricks.
• Experience handling large volumes of big data using Spark and the Hadoop ecosystem.
• Bachelor’s degree or foreign equivalent from an accredited institution, or equivalent progressive experience.

Preferred Skills

• Experience with Unity Catalog and scripting languages such as Python or Java.
• Familiarity with Agile/Scrum methodologies and coordinating with offshore engineering teams.
• Strong understanding of data structures, algorithms, and big data software technologies.

Key Skills

Education

Bachelor’s degree or foreign equivalent required from an accredited institution.