Description

You will design, build, and optimize data pipelines for terabyte-scale datasets, focusing on scalability and high-throughput performance.

This role is hybrid.

Responsibilities

• Develop and maintain containerized, serverless data platforms using Kubernetes.
• Lead migration efforts from EMR/EC2-based systems to Kubernetes-based architectures.
• Build and manage CI/CD pipelines using GitLab and Bitbucket.
• Optimize code performance for large-scale data processing and distributed systems.
• Collaborate on system design, requirements documentation, and GenAI integration initiatives.

Required Skills

• 5+ years of experience with Kubernetes, including deployment, migration, and infrastructure management.
• Strong proficiency in Python, SQL, or Scala for data engineering tasks.
• Experience with AWS cloud platforms and large-scale data processing (terabytes).
• Hands-on experience with CI/CD tools, specifically GitLab and Bitbucket.
• Knowledge of Big Data frameworks such as Spark or EMR.
• Familiarity with containerization, serverless architectures, and DevOps practices.
• Bachelor's degree in Computer Science or related field.

Preferred Skills

• AWS and/or Kubernetes certifications.
• Exposure to AI/GenAI tools, agents, or frameworks like Kiro or MCP.

Key Skills

Education

Bachelor's degree