Description

Design, build, and optimize containerized data pipelines and platforms for terabyte-scale datasets using Kubernetes and AWS.

This role is hybrid.

Responsibilities

• Design and optimize data pipelines processing terabyte-scale datasets with high throughput and scalability.
• Migrate existing EMR/EC2-based systems to Kubernetes-based serverless architectures.
• Build and maintain CI/CD pipelines using GitLab and Bitbucket for automated deployments.
• Develop and manage Kubernetes infrastructure, ensuring reliability and performance of distributed systems.
• Collaborate on system design and implementation, integrating GenAI tools and AI agents where applicable.

Required Skills

• 5+ years of experience with Kubernetes, including deployment, migration, and infrastructure management.
• Strong proficiency in at least one: Python, SQL, or Scala.
• Experience with AWS cloud platforms and containerization technologies.
• Hands-on experience with CI/CD pipelines and modern DevOps practices.
• Knowledge of distributed systems and big data frameworks (e.g., Spark, EMR).
• Familiarity with Elasticsearch and data processing optimization techniques.
• Experience with GitLab and Bitbucket for version control and pipeline management.

Preferred Skills

• Exposure to AI/GenAI tools, agents, or related frameworks.
• AWS and/or Kubernetes certifications.

Key Skills

Education

Any Gradute