Description

You will design and maintain scalable data pipelines using Databricks, PySpark, and Spark SQL for large-scale structured and unstructured datasets.

Responsibilities

• Build and optimize high-performance ETL/ELT workflows for data ingestion, cleaning, and aggregation.
• Implement Delta Lake, Delta Live Tables, and Databricks notebooks to ensure efficient data processing.
• Define data validation, quality checks, and normalization procedures to maintain governance standards.
• Mentor junior engineers and collaborate with data science and BI teams to support analytics initiatives.

Required Skills

• 6+ years of overall IT experience with 5+ years in Data Engineering or ETL development.
• Strong hands-on experience with Databricks, PySpark, and Apache Spark ecosystem.
• Expertise in Spark SQL, Delta Lake, and Unity Catalog.
• Proficient in writing intermediate-to-advanced SQL queries.
• Experience building data ingestion pipelines and large-scale data architectures.
• Familiarity with Agile methodologies including Scrum, Kanban, and SAFe.
• Ability to work with large-scale distributed data systems.

Preferred Skills

• Programming proficiency in Python and/or Scala.
• Experience with cloud platforms (AWS, Azure, or GCP) and serverless technologies.
• Familiarity with messaging systems like Kafka, Amazon MSK, or IBM MQ.

Key Skills

Education

Any Graduate