Description

You will design, develop, and maintain data pipelines using Databricks and Spark within an Azure environment.

Responsibilities

• Design and maintain data pipelines using Databricks, Spark, and cloud technologies.
• Optimize pipelines for performance, scalability, and reliability.
• Implement ETL processes, data warehouse solutions, and data validation methods.
• Troubleshoot data-related issues and perform root cause analysis.
• Collaborate with data scientists and analysts to meet technical data requirements.

Required Skills

• 5+ years of data engineering experience with Databricks and Spark.
• Proficiency in Python, PySpark, and SQL.
• Hands-on experience with Azure Data Factory (ADF).
• Experience with Azure Databricks Medallion Architecture, DLT, and Iceberg.
• Knowledge of Azure cloud platform and services.
• Understanding of data warehouse and data lake architectures.
• Bachelor's degree in Computer Science, Engineering, or equivalent work experience.

Preferred Skills

• Experience in a Financial or Corporate Banking context.
• Exposure to big data technologies such as Kafka, Hadoop, or Hive.

Key Skills

Education

Any Graduate