Description

You will build and scale lakehouse and AI data pipelines on Databricks.

Responsibilities

• Design, build, and maintain batch/streaming pipelines in Python and PySpark on Databricks (Delta Lake, Autoloader, Structured Streaming).
• Implement data models (Bronze/Silver/Gold), optimizing with partitioning, Z-ORDER, and indexing.
• Enable ML/AI workflows, including feature engineering, MLflow experiment tracking, and supporting RAG pipelines (embeddings, vector stores).
• Establish data quality checks (Great Expectations), lineage, and governance (Unity Catalog, RBAC).
• Champion CI/CD and IaC practices while troubleshooting performance and cost issues.

Required Skills

• 6+ years in data engineering with production pipeline experience.
• Expert in Python and PySpark (UDFs, Window functions, Spark SQL).
• Deep hands-on experience with Databricks: Delta Lake, Jobs/Workflows, Structured Streaming.
• Strong SQL and data modeling skills (dimensional, medallion, CDC).
• Experience enabling ML/AI workflows using MLflow and feature stores.
• Cloud proficiency on AWS, Azure, or GCP (object storage, IAM, networking).
• Familiarity with CI/CD tools (GitHub/GitLab/Azure DevOps) and testing (pytest).
• Experience with LLM workflows, including embeddings and vectorization.

Key Skills

Education

Any Graduate