Description

You will lead the migration of legacy Java data pipelines to Databricks and design scalable data solutions using Python, PySpark, and Delta Lake.

This role is hybrid.

Responsibilities

• Migrate legacy Java-based pipelines to Databricks and maintain scalable data pipelines using Apache Spark.
• Implement Medallion Architecture (Bronze, Silver, Gold) with Delta Lake to ensure reliable data delivery.
• Optimize Databricks jobs, queries, and Spark workloads for performance, scalability, and reliability.
• Mentor data engineers, lead proof-of-concept initiatives, and resolve complex data transformation issues.
• Design data models and infrastructure for efficient ETL/ELT processes and metadata management.

Required Skills

• Strong proficiency in Python and PySpark.
• Proven experience with Databricks and Apache Spark, including building and optimizing data pipelines.
• Deep knowledge of Delta Lake and performance analysis within Databricks environments.
• Strong data modeling skills and advanced SQL knowledge.
• Experience with batch and streaming data processing, including message queuing and scalable big data stores.
• 5+ years of experience in data engineering.
• Experience working with US-based clients in onsite/offshore delivery models.

Preferred Skills

• Experience with Talend.
• Knowledge of Java-based legacy systems.

Key Skills

Education

Any Graduate