Description

Lead the design and implementation of complex data solutions using Databricks and Azure. You will own the migration, governance, and orchestration of enterprise-grade data platforms.

Responsibilities

• Execute native Spark and Hadoop migrations to Databricks environments.
• Build data governance solutions using Unity Catalog and Starburst.
• Develop orchestration layers using Databricks workflows and Azure Data Factory.
• Implement CI/CD pipelines for Databricks within Azure DevOps.
• Extract logic from on-prem layers including SSIS, stored procedures, Informatica, and Vertica into PySpark.

Required Skills

• 5+ years of experience in data engineering roles.
• Expertise in PySpark and Apache Spark.
• Hands-on experience with Databricks, Azure, and AWS.
• Proficiency in SQL and Massive Parallel Processing (MPP) layers.
• Experience with Hadoop, MongoDB, and relational/non-relational data stores.
• Knowledge of ETL/ELT tools such as SSIS, Informatica, Matillion, and DBT.
• Experience with Apache Hudi and filesystem-based data extraction.
• Ability to build cloud-agnostic solutions.
• Strong background in data modeling, integration, and governance.

Key Skills

Education

Any Gradute