Description

Lead data engineering initiatives focused on scalable data workflows and technical direction.

This role is on-site.

Responsibilities

• Design and implement data pipelines using Azure Databricks and Apache Spark.
• Develop and optimize data ingestion processes using Kafka for real-time streaming.
• Integrate data sources across relational (DB2, PostgreSQL) and NoSQL (MongoDB) systems.
• Tune and scale production Spark jobs for optimal performance.
• Drive cross-functional delivery through technical influence.

Required Skills

• 15+ years in data engineering with 7+ years in a technical leadership role.
• Deep hands-on experience with Azure Databricks and Apache Spark, including Structured Streaming.
• Proficiency in Python for building scalable, reusable data workflows.
• Experience utilizing Kafka for real-time data ingestion and event streaming.
• Experience integrating with DB2, PostgreSQL, and MongoDB.
• Demonstrated expertise in performance tuning and scaling Spark jobs.

Preferred Skills

• Experience with NoSQL systems beyond MongoDB.
• Proven track record of leading technical direction in complex data environments.

Key Skills

Education

Any Graduate