Description

You will design, build, and maintain scalable data processes and parallel processing platforms on Azure.

Responsibilities

• Build and maintain distributed data pipelines using PySpark.
• Perform root cause analysis to identify and resolve performance bottlenecks in Spark jobs.
• Conduct deep-dive data analysis to troubleshoot and resolve data-related issues.
• Collaborate with stakeholders, IT, database engineers, and scientists to meet technical requirements.
• Implement algorithms within a distributed environment.

Required Skills

• 6-8 years of experience in data engineering.
• Strong proficiency in SQL.
• Extensive experience with PySpark and designing parallel processing platforms.
• Deep expertise in building data pipelines and distributed environments.
• Strong knowledge of Azure cloud services.
• Proven ability to perform deep-dive troubleshooting and performance tuning.

Preferred Skills

• Hands-on experience with Azure Synapse.
• Hands-on experience with Azure Data Factory.

Key Skills

Education

Not specified