Description

You will drive the migration of data environments from Apache Hive Metastore to Databricks Unity Catalog.

Responsibilities

• Analyze the current Hive Metastore environment to define a comprehensive migration strategy to Unity Catalog.
• Configure and manage Unity Catalog metastores, external locations, and credentials within Databricks workspaces.
• Develop and execute Scala-based scripts to migrate Hive Metastore tables, views, and associated metadata.
• Refactor existing Scala/Spark data pipelines to integrate seamlessly with Unity Catalog.
• Conduct thorough testing to validate data consistency, performance, and access control policies in the Unity Catalog environment.

Required Skills

• Strong proficiency in Scala for data manipulation and Spark development.
• In-depth knowledge of the Databricks platform, including Spark and Delta Lake.
• Hands-on experience with Unity Catalog setup, configuration, and migration strategies.
• Solid understanding of Hive Metastore concepts and integration patterns.
• Experience with Azure Cloud and its data storage services.
• Familiarity with data governance principles and access control mechanisms.
• Expertise in large-scale data migrations.

Key Skills

Education

Any Graduate