Description

You will design and implement end-to-end data engineering pipelines and lakehouse architectures on Azure and Snowflake.

This role is on-site.

Responsibilities

• Build scalable ETL/ELT pipelines using PySpark, Spark SQL, Python, and Talend.
• Architect and maintain Lakehouse structures with Delta Lake and Medallion layers.
• Implement real-time and batch data ingestion, including streaming with Spark Structured Streaming.
• Enforce data governance, access control, and lineage using Unity Catalog.
• Optimize Spark workloads through partitioning, caching, and cluster tuning to reduce cloud costs.

Required Skills

• 15+ years of experience in data engineering or data architecture.
• Strong expertise in Azure Databricks, Delta Lake, and Snowflake.
• Proficiency in Apache Spark (PySpark, Spark SQL), Python, and SQL.
• Hands-on experience with Azure Cloud services, specifically ADLS Gen2.
• Experience with ETL/ELT tools such as Talend.
• Knowledge of data modeling, CDC, and incremental loading techniques.
• Experience with CI/CD pipelines using Azure DevOps or Jenkins.
• Ability to work in Linux/Unix environments with shell scripting.

Preferred Skills

• Exposure to real-time streaming technologies like Kafka or Kinesis.
• Familiarity with workflow orchestration tools such as Airflow.

Key Skills

Education

Any Gradute