Description

Key Responsibilities

Leadership & Strategy

· Lead and mentor a team of data engineers.

· Define and implement enterprise data engineering best practices.

· Collaborate with stakeholders (business, analytics, data science) to translate requirements into technical designs.

· Drive architectural decisions for data platform modernization on Azure.

Data Engineering & Development

· Design and develop scalable data pipelines using Azure Data Factory (ADF).

· Build and optimize data transformations using Azure Databricks (PySpark / SQL / Scala).

· Implement ETL/ELT pipelines for structured and unstructured data.

· Integrate data from multiple sources (APIs, databases, files, streaming).

Data Architecture

· Design and maintain Lakehouse architecture using Databricks.

· Develop data models optimized for analytics (star/snowflake schemas).

· Implement Delta Lake for data consistency, reliability, and performance.

· Ensure data quality, lineage, governance, and security.

Performance & Optimization

· Tune Spark jobs for performance and cost efficiency.

· Optimize ADF pipelines for scheduling, monitoring, and throughput.

· Implement partitioning, caching, and data indexing strategies.

Cloud & DevOps

· Work with Azure services: ADLS Gen2, Synapse, Key Vault, Event Hub.

· Implement CI/CD pipelines (Azure DevOps, GitHub).

· Automate deployments using Infrastructure as Code (ARM/Bicep/Terraform).

· Monitor pipelines and workflows with logging and alerting.

---

Required Skills & Qualifications

Technical Skills

· Strong hands-on experience with:

   o Azure Databricks (PySpark, Spark SQL)

   o Azure Data Factory (ADF)

· Expertise in Python and SQL

· Experience with Delta Lake & Lakehouse architecture · Strong knowledge of:

   o Azure Data Lake Storage (ADLS Gen2)

   o Data warehousing concepts

   o ETL/ELT best practices

Data Engineering Skills

· Experience building large-scale distributed data pipelines · Proficiency with data modeling techniques · Familiarity with streaming frameworks (Kafka/Event Hub - nice to have)

Cloud & DevOps

· Experience with Azure ecosystem

· Exposure to:

   o Azure Synapse Analytics

   o CI/CD pipelines

   o Git version control

Leadership

· Proven experience leading teams or projects · Strong communication and stakeholder management skills

Key Skills

Education

Bachelor's degree