Description

You will build and maintain scalable data pipelines and Spark clusters on AWS infrastructure.

Responsibilities

• Deploy and configure Apache Spark clusters on AWS EKS, optimizing for performance and scalability.
• Develop and automate Spark data pipelines focusing on low latency and high performance.
• Design optimized SQL queries to improve data processing efficiency and minimize latency.
• Collaborate with data scientists and product managers to align technical implementations with project goals.
• Gather stakeholder requirements and document technical solutions clearly.

Required Skills

• 5+ years of data processing experience in large cloud-based infrastructure, specifically AWS.
• Strong proficiency in Python and PySpark for data pipeline development.
• Expert understanding of SQL, dimensional modeling, and analytical data warehouses like Snowflake.
• Experience with data pipeline orchestration tools such as Airflow.
• Knowledge of processing semi-structured file formats including JSON and Parquet.
• Hands-on experience with Python, SQL, AWS SDK, and DBT REST API.
• Proficiency with Git, Jenkins, and Terraform.

Preferred Skills

• Experience with Jinja, Shell scripting, and DBT.
• Development using serverless technologies such as AWS Glue, Lambda, or EMR.
• Familiarity with containers and infrastructure-as-code principles.

Key Skills

Education

Any Graduate