Description

Real-Time & Batch Integration: Configure and manage continuous data ingestion pipelines using Apache Kafka event streams for real-time market data, alongside scheduled batch processing of Parquet files and Delta Lake tables.

• Medallion Data Modeling: Architect and implement Databricks Lakehouse data models (Bronze/Silver/Gold layers) tailored for energy commodities, ensuring high performance, data lineage, scalability, and compute efficiency.

• Cross-Desk Reporting Delivery: Collaborate with Front, Middle, and Back Office stakeholders to transform raw transactional data into curated datasets supporting critical reporting for risk (VaR), PnL, scheduling, and regulatory compliance.

• Databricks Platform Architecture: Serve as a Subject Matter Expert in designing, configuring, and securing Databricks environments, including Unity Catalog governance, cluster optimization, CI/CD deployment workflows, and platform performance tuning.

• Data Engineering & Optimization: Build scalable ETL/ELT workflows using PySpark and SQL, optimizing large-scale distributed data processing across cloud-native lakehouse architectures.

• Governance & Reliability: Implement monitoring, alerting, and data quality frameworks to ensure reliable, auditable, and compliant enterprise data pipelines

Key Skills

Education

Any Graduate