Description

You will architect and manage Databricks Lakehouse platforms, delivering real-time and batch data pipelines for energy commodities and financial risk reporting.

This role is on-site.

Responsibilities

• Configure and maintain continuous data ingestion pipelines using Apache Kafka for real-time market data and scheduled batch processing of Parquet and Delta Lake tables.
• Design and implement Medallion architecture data models (Bronze/Silver/Gold) ensuring high performance, data lineage, and compute efficiency.
• Collaborate with Front, Middle, and Back Office stakeholders to transform raw transactional data into curated datasets for VaR, PnL, and regulatory compliance reporting.
• Architect, secure, and optimize Databricks environments, including Unity Catalog governance, cluster optimization, and CI/CD deployment workflows.
• Implement monitoring, alerting, and data quality frameworks to ensure reliable, auditable, and compliant enterprise data pipelines.

Required Skills

• 10+ years of experience in data engineering and platform architecture.
• Deep expertise with Databricks Lakehouse platform, including Unity Catalog and cluster optimization.
• Strong proficiency in building scalable ETL/ELT workflows using PySpark and SQL.
• Experience configuring and managing Apache Kafka for real-time event streams.
• Proven track record implementing Medallion data modeling architectures.
• Ability to design secure, governed, and high-performance cloud-native data solutions.
• Experience delivering data products for financial risk, PnL, or regulatory compliance reporting.

Preferred Skills

• Background in energy commodities or financial services data domains.
• Experience with CI/CD deployment workflows for data platforms.

Key Skills

Education

Any Graduate