Description

You will design, implement, and optimize large-scale data pipelines for capital markets environments. This role is hybrid.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks.
• Leverage PySpark, Spark, and SQL to transform and process large datasets.
• Integrate data from multiple sources, including Azure Blob Storage and ADLS.
• Optimize Databricks workloads for cost efficiency and performance.
• Implement and manage data security and governance standards using Unity Catalog.

Required Skills

• 10+ years of experience in data engineering.
• Strong expertise in Databricks, including Delta Lake, Unity Catalog, and Lakehouse Architecture.
• Proficiency in PySpark and Python.
• Expertise in SQL and Data Warehouse concepts.
• Hands-on experience with ETL/ELT pipelines.
• Familiarity with Azure Cloud Services.
• Experience working in capital markets data environments.

Key Skills

Education

Any Graduate