Description

You will design and develop Spark data processing pipelines for complex, large-scale transformations. You will own the performance tuning of these jobs to optimize processing time and resource usage. You will build and enhance common data processing frameworks focused on efficiency, scalability, and reusability. You will implement and optimize streaming data pipelines using Kafka and Flink. You will lead code reviews and provide technical mentorship to junior team members.

Responsibilities

• Design and develop Spark data processing pipelines for large-scale data transformations.
• Perform performance tuning on complex Spark jobs to optimize processing time and resource usage.
• Build and enhance common data processing frameworks focused on efficiency, scalability, and reusability.
• Implement and optimize streaming data pipelines using Kafka and Flink.
• Lead code reviews and provide technical mentorship to junior team members.

Required Skills

• 10+ years of professional software development experience.
• Expertise in Java and Spark for high-volume batch and streaming data processing.
• Hands-on experience with AWS services including SNS, SQS, EventBridge, Lambda, Glue, and Lake Formation.
• Proven ability to tune Spark job performance in large-scale environments.
• Experience with streaming technologies such as Kafka and Flink.
• Experience building common data processing frameworks or libraries.
• Proficiency with Docker and Kubernetes for containerization and orchestration.
• Strong background in MLOps, specifically managing offline, online, or in-line feature stores using SageMaker, Tecton, Feast, or Databricks.

Preferred Skills

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Any Graduate