Description

You will lead the design and implementation of complex data solutions and migration strategies within the Databricks ecosystem.

Responsibilities

• Migrate legacy Hadoop and native Spark workloads to Databricks.
• Build orchestration layers using Databricks workflows and Azure Data Factory.
• Develop CI/CD pipelines for Databricks using Azure DevOps.
• Extract logic from on-prem layers such as SSIS, stored procedures, Informatica, and Vertica into PySpark.

Required Skills

• 5+ years of experience in data engineering roles.
• Strong hands-on expertise in PySpark and Apache Spark.
• Experience with Massive Parallel Processing (MPP) in Spark SQL and PySpark.
• Proficiency in relational and non-relational data stores including SQL, Hadoop, and MongoDB.
• Experience with ETL/ELT tools such as SSIS, Informatica, Matillion, and DBT.
• Hands-on experience with Azure and AWS cloud environments.
• Knowledge of data modeling, integration, security, and governance.

Preferred Skills

• Process near real-time data using Auto Loader and Delta Live Tables (DLT).
• Design cost-effective infrastructure and security layers within Delta Lake.

Key Skills

Education

Any Graduate