Description

You will design, build, and optimize scalable data pipelines using Databricks, leveraging Apache Spark, Delta Lake, and Unity Catalog.

Responsibilities

• Participate in migrating approximately 20TB of on-premises Microsoft SQL Server data warehouse data to the Databricks platform.
• Convert and modernize hundreds of SQL Server tables and thousands of SSIS jobs to a modern architecture.
• Re-engineer SSIS ETL processes into Databricks notebooks, workflows, and orchestration frameworks.
• Implement data quality, validation, reconciliation, and audit controls throughout the migration process.
• Optimize performance and cost by using efficient Spark usage, partitioning strategies, and advanced query tuning.

Required Skills

• 4-7 years of overall data engineering experience.
• Hands-on experience with Databricks on AWS (preferred) or Azure.
• Strong proficiency in Apache Spark (PySpark and/or Scala) and SQL.
• Proven experience migrating on-premises SQL Server data warehouses to cloud platforms.
• Experience converting SSIS-based ETL pipelines into Spark-based solutions.
• Knowledge of Data Pipelines, ETL, and Data Warehousing principles.
• Familiarity with Performance Tuning techniques.

Key Skills

Education

Any Graduate