Description

Design and develop scalable ETL/ELT pipelines using Databricks, Delta Lake, and Databricks SQL.

This role is hybrid.

Responsibilities

• Develop batch, streaming, and CDC ingestion pipelines to enable near real-time data processing.
• Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data.
• Build and deploy pipelines using Databricks Asset Bundles (DAB) with automated CI/CD via GitHub or GitLab.
• Leverage Unity Catalog for secure data governance, access control, and lineage tracking.
• Integrate Power BI, Tableau, or Looker with Databricks using optimized connectors and security controls.

Required Skills

• 15+ years of experience in data engineering.
• Strong expertise in Databricks, Delta Lake, and Databricks SQL (DBX).
• Experience migrating data from Redshift, DB2, Oracle, MySQL, or PL/SQL into Databricks.
• Proficiency in building real-time streaming pipelines using Kafka and Structured Streaming.
• Strong SQL and data modeling skills with experience in Medallion architecture.
• Experience with CI/CD tools including GitLab.
• Knowledge of JDBC/ODBC connectors and distributed systems.

Preferred Skills

• Experience with Databricks Asset Bundles (DAB) for pipeline packaging and deployment.

Key Skills

Education

Bachelor's degree