Description

You will design, build, and maintain high-quality, scalable data pipelines.

Responsibilities

• Design, build, and optimize scalable pipelines in Databricks (PySpark, SQL, Delta Lake) for diverse data types.
• Ingest and process data from various sources including relational databases, APIs, PDFs, Excel, and web scraping outputs.
• Implement data quality frameworks, validation checks, and automated tests throughout the pipeline lifecycle.
• Own pipelines end-to-end, managing development through deployment, monitoring, and continuous improvement.
• Collaborate with data science and analytics teams to deliver model-ready datasets and maintain schema stability.

Required Skills

• 5+ years of professional experience in Data Engineering or a related field.
• Hands-on expertise with Databricks using PySpark, SQL, and Delta Lake.
• Proven track record building and deploying end-to-end production pipelines at scale.
• Experience handling structured and unstructured data, including PDF extraction, Excel, web scraping, and APIs.
• Strong foundation in data quality, testing, and observability practices.
• Ability to troubleshoot, debug, and optimize Spark/Databricks workloads for performance.
• Familiarity with dataset versioning and reproducibility concepts.
• Strong written and verbal communication skills for documentation and stakeholder alignment.

Preferred Skills

• Experience with AWS services.
• Familiarity with LangChain, LangGraph, or MCP for data processing.

Key Skills

Education

Any Graduate