Description

You will design, implement, and optimize large-scale data pipelines on Databricks. You own the end-to-end data architecture, from ingestion to consumption.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from Azure Blob Storage, ADLS, and relational/non-relational systems.
• Optimize Databricks workloads for cost efficiency, performance, and reliability.
• Implement data security, access controls, and governance using Unity Catalog.
• Deploy Databricks jobs and configurations using Asset Bundles and GitLab.

Required Skills

• 7+ years of experience in data engineering or architecture.
• Deep expertise in Databricks: Delta Lake, Lakehouse Architecture, Delta Live Pipelines, and Table Triggers.
• Proficiency in Python, SQL, and PySpark for big data processing.
• Strong knowledge of Master Data Management (MDM) principles.
• Experience with Azure Cloud Services and data warehouse concepts.
• Knowledge of Kafka for streaming data integration.
• Familiarity with Snowflake and GraphQL.
• Experience with Microservices architecture.

Preferred Skills

• Azure Data Engineer Associate or Databricks Certified Data Engineer Associate certification.
• Knowledge of Spark Streaming and real-time data solutions.

Key Skills

Education

Any Graduate