Description

You will design and implement scalable data pipelines on Databricks, owning the architecture from ingestion to governance.

Responsibilities

• Build production-grade pipelines using Apache Spark and Delta Lake, implementing medallion architecture patterns.
• Lead the rollout of Unity Catalog for data governance, access control, and lineage tracking.
• Tune Spark jobs for throughput and cost efficiency, utilizing Spark UI, query plans, and cluster metrics.
• Implement data quality standards using Great Expectations or DLT expectations and enforce security policies.
• Mentor junior engineers and collaborate with data scientists to productionize ML models via MLflow.

Required Skills

• 5+ years of data engineering experience with distributed systems (Spark, Hadoop).
• 3+ years of hands-on Databricks experience in production environments.
• Strong proficiency in Python (PySpark) and SQL; Scala experience is a plus.
• Deep understanding of Apache Spark internals: DAG execution, shuffle, memory management, and Catalyst optimizer.
• Extensive experience with Delta Lake, including ACID transactions, schema evolution, and time travel.
• Experience with cloud data platforms: AWS (S3, Glue), Azure (ADLS), or GCP (BigQuery, GCS).
• Familiarity with data orchestration tools such as Apache Airflow, Databricks Workflows, or dbt.
• Proficiency with version control (Git) and CI/CD practices for data engineering.

Preferred Skills

• Databricks Certified Associate or Professional certification (Data Engineer or Spark Developer).
• Experience with streaming architectures using Structured Streaming, Kafka, or Kinesis.

Key Skills

Education

Any Gradute