Description

Key Responsibilities

•       Design and implement end-to-end data engineering pipelines using Azure Databricks, ADLS Gen2, and Snowflake.

•       Develop scalable ETL/ELT pipelines using PySpark, Spark SQL, Python, and Talend.

•       Build and maintain Lakehouse architecture using Delta Lake and Medallion (Bronze, Silver, Gold) layers.

•       Implement real-time and batch data ingestion pipelines, including streaming using Spark Structured Streaming.

•       Design and enforce data governance, access control, and lineage using Unity Catalog.

•       Optimize Spark workloads through partitioning, caching, broadcast joins, and cluster tuning to improve performance and reduce cloud costs.

•       Architect and manage CI/CD pipelines using Azure DevOps, Jenkins, and Git for automated deployments.

•       Integrate multiple data sources and systems, ensuring high-quality, reliable, and scalable data delivery.

•       Collaborate with cross-functional teams including data analysts, scientists, and business stakeholders to support analytics and reporting needs.

•       Support data warehouse modernization initiatives, including migration from legacy systems to cloud platforms.

________________________________________

Required Qualifications

•       10+ years (ideally 15–20+) of experience in data engineering or data architecture.

•       Strong expertise in:

o       Databricks & Delta Lake

o       Snowflake Data Warehouse

o       Apache Spark (PySpark, Spark SQL)

•       Hands-on experience with Azure Cloud (ADLS Gen2, Azure Databricks, ADF).

•       Proficiency in Python and SQL for data engineering.

•       Experience with ETL/ELT tools such as Talend or Informatica.

•       Strong knowledge of data modeling, CDC (Change Data Capture), and incremental loading techniques.

•       Experience working in Linux/Unix environments with shell scripting.

________________________________________

Preferred Qualifications

•       Knowledge of data governance, compliance, and regulatory standards (e.g., IDMP).

•       Exposure to real-time data streaming technologies (Kafka, Kinesis).

•       Experience with multi-cloud environments (AWS, GCP).

•       Familiarity with workflow orchestration tools such as Airflow or Databricks Workflows.

________________________________________

Key Skills

•       Data Engineering & Architecture

•       Lakehouse & Data Warehousing

•       Spark Performance Optimization

•       Cloud Data Platforms (Azure)

•       ETL/ELT Pipeline Development

•       Data Governance & Security

•       CI/CD & DevOps Practices

________________________________________

Education

•       Bachelor’s degree in Information Technology, Computer Science, or a related field.

________________________________________

Nice-to-Have Traits

•       Strong analytical and problem-solving skills

•       Ability to work in enterprise-scale, complex environments

•       Experience working with global stakeholders and cross-functional teams

•       Leadership capability with mentoring experience

Key Skills

Education

Any Gradute