Description

Migrate Python-based ETL jobs and Looker views to Spark ETL jobs on Databricks while ensuring data quality across all pipelines.

This role is on-site.

Responsibilities

• Migrate existing Python ETL jobs and Looker views to Spark ETL jobs running on Databricks.
• Update Data Lake tables via regular ETL job execution.
• Collaborate with the team to perform QA on new tables.
• Ensure data quality across all migrated pipelines.

Required Skills

• 4+ years of experience in data engineering.
• Proficiency with PySpark and Python.
• Hands-on experience with Databricks.
• Strong SQL skills for data manipulation.
• Experience working within AWS environments.
• Understanding of Data Warehouse (DWH) concepts.

Preferred Skills

• Bachelor's degree in Computer Science.
• Effective communication skills.

Key Skills

Education

Bachelor's degree in Computer Science