Description

Lead the design and development of end-to-end data pipelines within Databricks and Snowflake environments.

Responsibilities

• Design and implement end-to-end data ingestion and transformation processes.
• Execute data migrations from SQL Server and Oracle to Databricks.
• Develop complex queries, functions, and ETL logic using Scala and Spark SQL.
• Manage data lake architecture using Delta tables and various data warehousing patterns.
• Integrate multiple source systems into unified data environments.

Required Skills

• 4+ years of experience in data engineering.
• Extensive development experience in Databricks using Scala.
• Strong proficiency in Spark SQL.
• Deep understanding of data warehousing concepts, including ETL and SCD types.
• Hands-on experience with Snowflake.
• Experience working with SQL Server or Oracle databases.
• Expertise in writing complex queries and functions using T-SQL or PL/SQL.
• Practical knowledge of data lake and Delta table concepts.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Big Data Hadoop ecosystems.

Key Skills

Education

Bachelor's degree