Description

Lead the migration of Hadoop and HDFS workloads to Databricks, owning the end-to-end conversion of legacy environments.

This role is remote.

Responsibilities

• Migrate large-scale datasets and Hive tables to Databricks, converting legacy MapReduce, Hive, and Spark jobs to PySpark.
• Build and maintain ETL/ELT pipelines using Databricks, implementing Delta Lake and Lakehouse solutions.
• Develop data solutions using Python, PySpark, Spark SQL, and SQL, performing source-to-target mapping and transformation.
• Validate migrated data for accuracy and completeness, troubleshooting pipeline issues and optimizing Spark performance.
• Document migration processes and support production deployment and post-migration validation with engineering teams.

Required Skills

• 8+ years of Data Engineering or Big Data experience.
• Strong hands-on experience with Hadoop, HDFS, Hive, and Apache Spark.
• Proven experience migrating Hadoop workloads to Databricks.
• Proficiency in Python, PySpark, Spark SQL, and SQL.
• Experience developing ETL/ELT pipelines and implementing Delta Lake.
• Experience with data validation, reconciliation, and Spark performance tuning.
• Proficiency with Git and CI/CD workflows.

Preferred Skills

• Experience with Cloudera or Hortonworks to Databricks migration.
• Familiarity with Unity Catalog, Databricks Workflows, Auto Loader, or Delta Live Tables.
• Cloud experience with AWS, Azure, or GCP.

Key Skills

Education

Bachelor's degree