Description

You will lead the migration of data processing ecosystems from Hadoop to Databricks on GCP.

This role is remote.

Responsibilities

• Assess existing Hadoop infrastructure, including Spark and MapReduce jobs, to plan the migration to Databricks.
• Refactor Spark jobs and rewrite MapReduce jobs using Spark DataFrame/Dataset APIs for Databricks compatibility.
• Update Java and Scala codebases to comply with Databricks runtime environments.
• Develop unit and integration tests to ensure data parity and performance consistency between systems.
• Fine-tune Spark configurations and optimize data ingestion and transformation processes.

Required Skills

• 6+ years of experience in Data Engineering focusing on ETL, data pipelines, and data platforms.
• Expertise in Databricks, including Spark on Databricks and Delta Lake.
• 5+ years of experience in the Hadoop ecosystem, specifically Spark and MapReduce.
• 5+ years of proficiency in Scala and Java.
• Advanced SQL knowledge.
• Strong understanding of Spark internals and optimization techniques.
• Experience with data modeling, data governance, and enterprise-scale lakehouse platforms.
• Experience using test frameworks such as Great Expectations.

Preferred Skills

• Certified Databricks Engineer.

Key Skills

Education

Any Graduate