Description

You will design and build the inference service platform that serves as the backbone for AI models at scale.

Responsibilities

• Own and architect core platform components including the AI Gateway, Resource Orchestrator, Runtime Engines, and Autoscaler systems.
• Design highly modular, scalable, and extensible low-level designs for inference infrastructure components and establish architectural patterns.
• Build robust deployment pipelines for seamless model updates with zero-downtime deployments and intelligent routing systems.
• Implement highly performant and optimized software for low-latency, high-throughput inference serving in distributed systems.
• Build high-performance telemetry and observability stack for inference metrics, performance tracking, and comprehensive debugging capabilities.

Required Skills

• 5+ years of experience building and scaling backend systems, distributed platforms, or inference infrastructure.
• Strong understanding of AI/ML inference systems.
• Golang
• Kubernetes
• Docker
• Python
• Redis
• PostgreSQL
• Apache Kafka
• Microservices
• Load Balancing
• API Gateway

Preferred Skills

• Experience with inference engines like vLLM, TGI, or TensorRT-LLM.
• Familiarity with distributed systems architecture and autoscaling strategies.

Key Skills

Education

Any Graduate