Description

Key Responsibilities

Design, develop, and optimize data pipelines handling terabyte-scale datasets

Work with complex algorithms to process and analyze large volumes of data efficiently

Optimize code performance for scalability and high-throughput systems

Build and maintain containerized, serverless data platforms using Kubernetes

Support migration efforts from EMR/EC2-based systems to Kubernetes-based architecture

Maintain and enhance existing Kubernetes infrastructure

Develop and manage CI/CD pipelines using tools like GitLab and Bitbucket

Collaborate on requirements documentation, system design, and implementation

Contribute to emerging initiatives involving:

GenAI integration

AI agents and automation frameworks

Technologies such as Kiro (Amazon GenAI) and MCP (Model Context Protocol)

Technical Environment

Cloud Platforms: AWS (current), with exposure to Google Cloud and open-source ecosystems

Core Technologies:

Kubernetes (primary focus)

Elasticsearch

Big Data frameworks (EMR, distributed systems)

Programming Languages: Python, SQL, Scala (flexible)

DevOps & Tooling: GitLab, Bitbucket, CI/CD pipelines

Required Skills & Experience

Strong experience with Kubernetes, including deployment, migration, and infrastructure management

Experience working with large-scale data (terabytes) and distributed systems

Proficiency in at least one: Python, SQL, or Scala

Understanding of data processing optimization techniques

Familiarity with cloud-native architectures and containerization

Experience with CI/CD pipelines and modern DevOps practices

Exposure to AI/GenAI tools, agents, or related frameworks is a strong plus

AWS and/or Kubernetes certifications preferred

Key Skills

Education

Bachelor's degree