Description

Lead data engineering initiatives focused on scalable data workflows.

Responsibilities

• Design and implement data pipelines using Azure Databricks and Apache Spark.
• Develop and optimize data ingestion processes using Kafka for real-time streaming.
• Integrate data sources across relational (DB2, PostgreSQL) and NoSQL (MongoDB) systems.
• Tune and scale production Spark jobs for optimal performance.
• Lead technical direction and drive cross-functional delivery through influence.

Required Skills

• 15+ years in data engineering with 7+ years in a technical leadership role.
• Deep hands-on experience with Azure Databricks and Apache Spark, including Structured Streaming.
• Proficiency in Python for building scalable, reusable data workflows.
• Experience utilizing Kafka for real-time data ingestion and event streaming.
• Experience integrating with DB2, PostgreSQL, and MongoDB.
• Demonstrated expertise in performance tuning and scaling Spark jobs.
• Any Graduate education required.

Key Skills

Education

Any Graduate