Description

Drive the migration of data environments from Apache Hive Metastore to Databricks Unity Catalog.

This role is on-site.

Responsibilities

• Analyze the current Hive Metastore environment to define and execute a comprehensive migration strategy.
• Configure and manage Unity Catalog metastores, external locations, and credentials within Databricks workspaces.
• Develop and execute Scala-based scripts to migrate Hive Metastore tables, views, and associated metadata.
• Refactor existing Scala/Spark data pipelines to integrate seamlessly with Unity Catalog.
• Conduct thorough testing to validate data consistency, performance, and access control policies.

Required Skills

• 7+ years of experience in data engineering or migration roles.
• Strong proficiency in Scala for data manipulation and Spark development.
• In-depth knowledge of the Databricks platform, including Spark and Delta Lake.
• Hands-on experience with Unity Catalog setup, configuration, and migration strategies.
• Solid understanding of Hive Metastore concepts and integration patterns.
• Experience with Azure Cloud and its data storage services.
• Familiarity with data governance principles and access control mechanisms.
• Proven expertise in executing large-scale data migrations.

Key Skills

Education

Any Graduate