Description

You will design and implement performant data ingestion dataflows using Azure Databricks.

Responsibilities

• Design and implement dataflows from multiple sources using Azure Databricks.
• Execute programming, modelling, and data integration to support automation and DevOps practices.
• Perform unit testing and large-scale testing to ensure required performance.
• Mentor other team members on technical tasks.

Required Skills

• Years of experience in Spark using Scala or Python.
• Proficiency in PySpark and Scala development on Azure Databricks.
• Experience with end-to-end pipeline development on Azure Data Factory.
• Intermediate SQL expertise.
• Hands-on experience with Azure Storage, Azure Data Lake, and Azure SQL Data Warehouse.
• Knowledge of NoSQL databases such as MongoDB or Cosmos.
• Experience with Azure Event Hubs.
• Proficiency with Git for code versioning.
• Practical experience with Continuous Integration and deployment (CI/CD) tools.

Preferred Skills

• Experience with Jenkins builds and release pipelines.

Key Skills

Education

Any graduate