Description

Design and optimize large-scale Spark data processing pipelines for batch and streaming workloads.

This role is on-site.

Responsibilities

• Design and develop Spark data processing pipelines for complex, large-scale transformations.
• Perform performance tuning on complex Spark jobs to optimize processing time and resource usage.
• Build and enhance common data processing frameworks focused on efficiency, scalability, and reusability.
• Implement and optimize streaming data pipelines using Kafka and Flink.
• Lead code reviews and provide technical mentorship to junior team members.

Required Skills

• 10+ years of professional software development experience.
• Expertise in Java and Spark for high-volume batch and streaming data processing.
• Hands-on experience with AWS services including SNS, SQS, EventBridge, Lambda, Glue, and Lake Formation.
• Proven ability to tune Spark job performance in large-scale environments.
• Experience with streaming technologies such as Kafka and Flink.
• Experience building common data processing frameworks or libraries.
• Proficiency with Docker and Kubernetes for containerization and orchestration.
• Strong background in MLOps, specifically managing offline, online, or in-line feature stores using SageMaker, Tecton, Feast, or Databricks.

Preferred Skills

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Any Graduate