Description

Lead end-to-end data engineering projects focused on Databricks development and complex data migrations.

Responsibilities

• Design and develop end-to-end data ingestion and transformation pipelines using Scala within Databricks.
• Execute data migrations from legacy SQL Server and Oracle systems to Databricks environments.
• Implement data warehousing solutions utilizing ETL concepts and various Slowly Changing Dimension (SCD) types.
• Manage data lake architectures and delta table implementations.
• Write complex queries and functions to handle data from multiple source systems.

Required Skills

• 5+ years of experience in data engineering.
• Extensive development and design experience in Databricks using Scala.
• Strong proficiency in Scala programming for data processing.
• Hands-on experience with Spark SQL.
• Deep understanding of data warehousing, ETL concepts, and SCD types.
• Expertise in data lake and delta table concepts.
• Experience with SQL Server or Oracle databases.
• Proficiency in writing complex queries using T-SQL or PL/SQL.
• Bachelor's degree.

Preferred Skills

• Experience with Snowflake.
• Background in Big Data Hadoop environments.

Key Skills

Education

Bachelor's degree