Description

Build and maintain large-scale ML pipelines and feature engineering systems on AWS SageMaker and Hadoop infrastructure.

This role is hybrid.

Responsibilities

• Optimize feature engineering pipelines using PySpark, Pandas, and PyArrow on Hadoop-based infrastructure.
• Build near real-time ML pipelines using Apache Kafka and Spark Streaming.
• Refactor and modularize ML codebases to enhance reusability, maintainability, and performance.
• Integrate with model serving frameworks to support testing, deployment, and rollback processes.
• Monitor and troubleshoot production ML pipelines to ensure high reliability, low latency, and cost efficiency.

Required Skills

• 5+ years of experience with Python and PySpark.
• Strong proficiency with Apache Kafka for streaming data.
• Experience with Hadoop Hive and Hadoop-based infrastructure.
• Expertise in Pandas for data manipulation and feature engineering.
• Familiarity with AWS SageMaker MLOps ecosystem.
• Experience with PyArrow for efficient data interchange.
• Ability to collaborate with platform teams on compute capacity planning and resource allocation.

Preferred Skills

• Experience contributing to internal ML platforms and documenting best practices.

Key Skills

Education

Any Graduate