Description

Role Overview

We are looking for an experienced and results-driven Data Engineer to join our growing Data Engineering practice. The ideal candidate will be proficient in building scalable, high-performance data transformation pipelines using Snowflake, Azure Databricks, and Matillion, and will thrive in fast-paced, client-facing consulting environment. In this role, you will be instrumental in ingesting, transforming, and delivering high-quality data to enable data-driven decision-making across our clients' organizations — owning your deliverables like a true Tech Doctor.

Key Responsibilities

• Design, configure, and optimize ingestion, transformation, and orchestration workflows using Matillion DPC where applicable.
• Design and implement scalable ELT pipelines using dbt on Snowflake, following industry-accepted best practices.
• Build and maintain data processing workloads on Azure Databricks (PySpark/Spark SQL), including notebooks, jobs, Delta Lake tables, and Lakehouse/medallion architectures.
• Build ingestion pipelines from various sources including relational databases, APIs, cloud storage, and flat files into Snowflake and Databricks (Delta Lake).
• Implement data modelling and transformation logic to support layered architecture (staging, intermediate, and mart layers, or medallion architecture) to enable reliable and reusable data assets.
• Leverage orchestration tools (e.g., Airflow, dbt Cloud, Azure Data Factory, or Databricks Workflows) to schedule and monitor data pipelines.
• Apply CI/CD and Git-based workflows for version-controlled deployments.
• Write well-documented, maintainable code using Git for version control and CI/CD processes.

Required Qualifications

• Mandatory: Hands-on experience with Azure Databricks — developing and deploying pipelines using PySpark/Spark SQL, Delta Lake, notebooks, clusters, and Databricks Workflows/Jobs in a production environment.
• Hands-on experience with Matillion Data Productivity Cloud (Matillion DPC) for data ingestion, transformation, or orchestration.
• Expert-level SQL and strong understanding of ELT principles; strong understanding of ELT patterns and data modelling (Kimball/Dimensional preferred).
• Experience with Git, CI/CD, and deployment workflows in a team setting.
• Familiarity with orchestrating workflows using tools like dbt Cloud, Airflow, Azure Data Factory, or Databricks Workflows.
• Comfort with ambiguity, competing priorities, and fast-changing client environments — we move fast and adapt faster.
• Passion for continuous learning and knowledge sharing (tech talks, meetups, internal upskilling).

Requirements added by the job poster

• 6+ years of work experience with PySpark

• 2+ years of work experience with Microsoft Azure

Key Skills

Education

Any Graduate