Description

Own the development, optimization, and reliability of large-scale machine learning pipelines and infrastructure.

This role is hybrid.

Responsibilities

• Optimize and maintain feature engineering pipelines using PySpark, Pandas, and PyArrow on Hadoop-based infrastructure.
• Build near real-time ML pipelines using Apache Kafka and Spark Streaming.
• Refactor and modularize ML codebases to enhance reusability, maintainability, and performance.
• Monitor and troubleshoot production ML pipelines to ensure high reliability, low latency, and cost efficiency.
• Collaborate with platform teams on compute capacity planning, resource allocation, and system upgrades.

Required Skills

• 5+ years of experience with Python.
• Strong proficiency in Apache Kafka for streaming data.
• Experience with Hadoop, Hive, and distributed storage.
• Hands-on expertise with PySpark and Spark Streaming.
• Proficiency in Pandas and PyArrow for data manipulation.
• Experience with AWS SageMaker MLOps ecosystem.
• Ability to integrate with existing model serving frameworks for deployment and rollback.

Preferred Skills

• Experience contributing to internal ML platforms and documenting best practices.
• Background in system architecture for high-throughput data pipelines.

Key Skills

Education

Any Graduate