Description

Lead the design, development, and maintenance of data pipelines using Databricks and Spark within an Azure environment.

Responsibilities

• Design and optimize data pipelines for performance, scalability, and reliability.
• Implement ETL processes and data warehouse solutions while ensuring compliance with data governance and security policies.
• Troubleshoot data-related issues and provide root cause analysis and recommendations.
• Create data validation methods and document pipeline specifications and enhancements.
• Collaborate with data scientists and analysts to meet organizational data requirements.

Required Skills

• 5+ years of experience in data engineering focusing on Databricks and Spark.
• Deep understanding of distributed computing and Spark for data processing.
• Expertise in Python, PySpark, and SQL for database querying and processing.
• Hands-on experience with Azure Databricks Medallion Architecture and DLT.
• Proficiency with Azure Data Factory and the broader Azure cloud platform.
• Strong knowledge of data warehouse and data lake architectures.
• Experience managing onshore-offshore coordination.
• Experience with Iceberg.
• Bachelor's degree in Computer Science, Engineering, or equivalent work experience.

Preferred Skills

• Experience in Financial or Corporate Banking contexts.
• Familiarity with Kafka, Hadoop, or Hive.

Key Skills

Education

Any Graduate