Description

Design and develop ETL pipelines to ingest, transform, and validate data for business intelligence and analytics teams. You will own the data pipeline architecture within AWS cloud environments, ensuring optimal performance and cost efficiency.

Responsibilities

• Design and develop ETL pipelines for collecting, validating, and transforming data based on business specifications.
• Maintain optimal data pipeline architecture and design ETL jobs for execution within AWS cloud environments.
• Develop automated unit, functional, and performance tests to ensure data quality and system reliability.
• Optimize ETL workloads to reduce processing time and cloud costs.
• Participate in peer code reviews and provide support for production operations.

Required Skills

• 5+ years of experience in Java or Scala.
• 5+ years of experience in ETL projects.
• 3+ years of experience in big data projects.
• Strong experience with AWS EMR, AWS EKS, and Apache Spark.
• Strong experience with serverless technologies including AWS DynamoDB and AWS Lambda.
• Proficiency in processing JSON files and writing complex SQL queries.
• Experience with columnar storage formats like ORC or Parquet and various compression techniques.
• Experience with Unix shell scripting and unit testing using JUnit or ScalaTest.
• Proficiency with Git, Maven, or Gradle.

Preferred Skills

• AWS Step Functions or Cloud BPM.
• Python scripting and hands-on DevOps skills.

Key Skills

Education

Any Graduate