Description

You will own the development and optimization of ETL pipelines using PySpark and Databricks, supporting data migration initiatives. Your work will bridge legacy on-premise systems with cloud-based solutions.

Responsibilities

• Develop and optimize PySpark ETL pipelines for data migration projects.
• Migrate legacy Informatica workflows to Databricks cloud environments.
• Cleanse and transform structured and unstructured data from diverse vendors like Bloomberg.
• Implement business rules to ensure data accuracy and reliability in models.
• Collaborate within an Agile team to accelerate development cycles.

Required Skills

• 5+ years of proficiency in PySpark for data processing and pipeline development.
• Hands-on experience with Databricks workflows and cloud data environments.
• Strong background in data cleansing, hierarchy management, and applying business rules.
• Experience with Informatica and migration to cloud-based solutions.
• Proven ability in data modeling and related development fields.
• Effective collaboration skills within a small, Agile team.

Key Skills

Education

Any Graduate