Description

Build and optimize scalable data pipelines using Spark and the Azure ecosystem.

This role is on-site.

Responsibilities

• Develop and maintain data processing workflows using Spark and Databricks.
• Write scalable, production-grade code using Python.
• Optimize Spark jobs and SQL queries for performance.
• Debug complex data issues using pandas and API debugging techniques.

Required Skills

• 5-7 years of experience as a Spark and Databricks developer.
• Proficiency in Python and writing scalable code.
• Experience with Azure components including Azure Databricks, ADLS Storage, Key Vault, and Azure SQL.
• Knowledge of at least one Python framework such as Django, Flask, or Pyramid.
• Experience with Active Directory Authentication.
• Hands-on experience with Azure Git.
• Familiarity with Visual Studio Code and testing tools for unit, functional, and load testing.

Preferred Skills

• Knowledge of Delta Lake and Unity Catalog.
• Understanding of Azure DevOps and ability to modify Azure ARM template scripts.

Key Skills

Education

Any Graduate