Description

You will design and develop scalable ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader, and DLT.

This role is hybrid.

Responsibilities

• Build batch, streaming, and CDC ingestion pipelines for near real-time data processing.
• Migrate data from Redshift, DB2, Oracle, MySQL, and PL/SQL into Databricks.
• Implement ingestion patterns with Auto Loader, including checkpointing and schema evolution.
• Develop reusable DBX functions for transformation, data quality checks, and validation.
• Manage Unity Catalog for governance, lineage, and secure data sharing via Delta Sharing.

Required Skills

• 13+ years of experience in data engineering and distributed systems.
• Expertise in Databricks, Delta Lake, and Databricks Asset Bundles (DAB).
• Strong SQL and data modeling skills with experience in Medallion architecture.
• Hands-on experience with Kafka and Structured Streaming for real-time pipelines.
• Proficiency in optimizing Spark jobs and SQL queries.
• Experience integrating Power BI, Tableau, or Looker with Databricks.
• Automated CI/CD processes using GitHub or GitLab.

Preferred Skills

• Experience with structured and semi-structured data ingestion.
• Knowledge of external partners and cross-domain data sharing.

Key Skills

Education

Bachelor's degree