Description

You will design, develop, and optimize data pipelines for terabyte-scale datasets while migrating infrastructure from EMR/EC2 to Kubernetes.

This role is hybrid.

Responsibilities

• Build and maintain containerized, serverless data platforms using Kubernetes and optimize for scalability.
• Develop and manage CI/CD pipelines using GitLab and Bitbucket to support automated deployments.
• Process and analyze large volumes of data using distributed systems, focusing on performance and high-throughput.
• Collaborate on system design, requirements documentation, and implementation of emerging GenAI and AI agent initiatives.
• Support migration efforts from legacy EMR/EC2 architectures to modern Kubernetes-based solutions.

Required Skills

• 5+ years of experience with Kubernetes, including deployment, migration, and infrastructure management.
• Proficiency in Python, SQL, or Scala for data processing and pipeline development.
• Strong understanding of AWS cloud-native architectures and distributed systems.
• Experience building and optimizing data pipelines for terabyte-scale datasets.
• Familiarity with CI/CD tools such as GitLab or Bitbucket.
• Knowledge of Big Data frameworks like Spark or EMR.
• Bachelor’s degree or equivalent experience (Any Graduate).

Preferred Skills

• AWS and/or Kubernetes certifications.
• Exposure to AI/GenAI tools, agents, or frameworks such as Kiro or MCP.

Key Skills

Education

Any Gradute