Description

Design, build, and maintain scalable data pipelines in Databricks for diverse data types.

This role is hybrid.

Responsibilities

• Build and optimize scalable pipelines using Databricks, PySpark, SQL, and Delta Lake.
• Ingest and process structured and unstructured data from relational databases, APIs, PDFs, Excel, and web scraping outputs.
• Implement data quality frameworks, validation checks, and automated tests throughout the pipeline lifecycle.
• Own pipelines end-to-end, managing development, deployment, monitoring, and continuous improvement.
• Collaborate with data science and analytics teams to deliver model-ready datasets and maintain schema stability.

Required Skills

• 5+ years of professional experience in Data Engineering or a related field.
• Hands-on expertise with Databricks, PySpark, SQL, and Delta Lake.
• Proven track record building and deploying end-to-end production pipelines at scale.
• Experience handling structured and unstructured data, including PDF extraction, Excel, web scraping, and APIs.
• Strong foundation in data quality, testing, and observability practices.
• Ability to troubleshoot, debug, and optimize Spark/Databricks workloads for performance.
• Familiarity with dataset versioning and reproducibility concepts.

Preferred Skills

• Experience with AWS services.
• Familiarity with LangChain, LangGraph, or MCP for data processing.

Key Skills

Education

Any Graduate