Description

Develop and optimize ETL pipelines using PySpark and Databricks to support data migration projects. Migrate legacy on-premise applications from Informatica to cloud-based solutions. Cleanse and transform unstructured and structured data from multiple vendors, including Bloomberg. Implement data rules based on business requirements to create accurate and reliable data models. Collaborate with team members in an Agile environment to accelerate development.

Responsibilities

• Develop and optimize ETL pipelines using PySpark and Databricks.
• Migrate legacy on-premise applications from Informatica to cloud-based solutions.
• Cleanse and transform unstructured and structured data from multiple vendors.
• Implement data rules based on business requirements to create accurate data models.
• Collaborate with team members in an Agile environment to ensure project success.

Required Skills

• Proficiency in PySpark with a focus on data processing and pipeline development.
• Experience with Databricks workflows and cloud-based data environments.
• Strong understanding of data cleansing, hierarchy, and applying business rules.
• Background in development, data modeling, or related fields.
• Ability to work effectively in a small, collaborative team.
• 5+ years of professional experience.
• Any Graduate degree.

Preferred Skills

• Experience with Informatica legacy systems.
• Work with data sources such as Bloomberg.

Key Skills

Education

Any Graduate