Description

You will build and maintain scalable data infrastructure and high-performance processing pipelines on AWS.

Responsibilities

• Deploy and configure Apache Spark clusters on AWS EKS, optimizing for resource allocation and scalability.
• Develop and automate Spark data pipelines focused on low latency and high performance.
• Design optimized SQL queries to improve data processing efficiency and minimize system latency.
• Collaborate with data scientists and product managers to align technical implementations with project goals.
• Document technical solutions and gather requirements from stakeholders to ensure platform resilience.

Required Skills

• 5+ years of data processing experience in large-scale cloud infrastructure, specifically AWS.
• Hands-on software development experience in Python and PySpark.
• Expert proficiency in SQL and dimensional modeling.
• Experience with analytical data warehouses like Snowflake.
• Proficiency with data pipeline orchestration using Airflow.
• Experience processing semi-structured file formats such as JSON or Parquet.
• Strong understanding of data engineering best practices for production workloads.
• Bachelor’s degree in Computer Science, Data Science, or a related field.

Preferred Skills

• Experience with DBT, Jinja, and Shell scripting.
• Development using serverless technologies like AWS Glue, Lambda, or EMR.
• Familiarity with containers, Infrastructure-as-Code, Terraform, Jenkins, and Git.

Key Skills

Education

Any Graduate