Description

Design and implement large-scale data processing pipelines using Hadoop and Spark ecosystems.

Responsibilities

• Build and maintain data processing pipelines using MapReduce and Spark constructs like Joins.
• Write optimized SQL queries to ensure high performance across RDBMS and No-SQL architectures.
• Develop and execute comprehensive unit tests covering both positive and negative scenarios.
• Debug cloud infrastructure issues using AWS CloudWatch.
• Collaborate within an Agile environment to deliver data engineering solutions.

Required Skills

• 5-6 years of hands-on experience with Hadoop, Spark (Scala or Python), HBase, Hive, and Sqoop.
• Proficiency in Python or Scala for data engineering tasks.
• Working knowledge of AWS services including Lambda, EC2, ECS, EMR, Athena, and S3.
• Experience with Git or Bitbucket for version control.
• Strong understanding of RDBMS and No-SQL database architectures.
• Ability to write efficient, optimized SQL.
• Experience debugging in AWS CloudWatch.
• Bachelor's degree in Computer Science Engineering or an equivalent field.

Preferred Skills

• Experience with Google Data Engineering.

Key Skills

Education

Bachelor's degree