Description

You will lead the migration of a 20TB on-premises Microsoft SQL Server data warehouse to Databricks, re-engineering SSIS ETL processes into modern Spark-based solutions.

This role is on-site.

Responsibilities

• Migrate 20TB of SQL Server data to Databricks on AWS or Azure.
• Convert hundreds of SQL Server tables and thousands of SSIS jobs to Delta Lake and Unity Catalog.
• Re-engineer SSIS ETL pipelines into Databricks notebooks, workflows, and orchestration frameworks.
• Implement data quality, validation, reconciliation, and audit controls.
• Optimize performance and cost using efficient Spark usage, partitioning strategies, and query tuning.

Required Skills

• 4-7 years of overall data engineering experience.
• Hands-on experience with Databricks on AWS (preferred) or Azure.
• Strong proficiency in Apache Spark (PySpark and/or Scala) and SQL.
• Proven experience migrating on-premises SQL Server data warehouses to cloud platforms.
• Experience converting SSIS-based ETL pipelines into Spark-based solutions.
• Knowledge of Data Pipelines, ETL, and Data Warehousing principles.
• Familiarity with Performance Tuning techniques.

Preferred Skills

• Experience with Unity Catalog for governance.
• Background in large-scale data warehousing projects.

Key Skills

Education

Any Graduate