Description

You will develop and maintain an inference platform for serving large language models optimized for various GPU platforms.

Responsibilities

• Work on complex AI and cloud engineering projects across the full product development lifecycle, from ideation to operations.
• Build tooling and observability to monitor system health and develop auto-tuning capabilities.
• Construct benchmarking frameworks to assess model serving performance and guide infrastructure tuning.
• Build native cross-platform inference support across NVIDIA and AMD GPUs for diverse model architectures.
• Contribute to open-source inference engines to improve performance on DigitalOcean cloud.

Required Skills

• 5+ years of experience in building services on distributed systems using containerization (Kubernetes, Docker).
• Deep experience with CI/CD pipelines, APIs, authentication, and authorization.
• Experience hosting and running inference for Large Language Models (LLMs).
• Familiarity with inference engines such as vLLM, SGLang, and Modular Max.
• Experience with distributed inference serving frameworks like llm-d, NVIDIA Dynamo, and Ray Serve.
• Experience with AMD and NVIDIA GPUs, including CUDA, ROCm, AITER, NCCL, and RCCL.
• Knowledge of distributed inference optimization techniques (e.g., tensor/data parallelism, KV cache optimizations).
• Proficiency with cloud environments and infrastructure as code practices.
• Strong verbal and written communication skills for technical documentation.

Key Skills

Education

Bachelor's degree