Description

You will build and manage big data pipelines within an AWS environment, owning migrations from on-prem HDFS to S3 and optimizing EMR and S3 I/O performance.

This role is on-site.

Responsibilities

• Execute migrations of data from on-prem HDFS to S3.
• Optimize performance within EMR clusters and S3 I/O operations.
• Develop data processing workflows using PySpark and SQL.
• Manage orchestration and data storage using MWAA and Iceberg.

Required Skills

• 6+ years of experience in big data development.
• 3+ years of hands-on experience with AWS.
• Proficiency with EMR and S3.
• Experience with Apache Iceberg.
• Strong PySpark and SQL skills.
• Experience with MWAA.
• Proven track record with HDFS to S3 migration scenarios.
• Ability to perform EMR performance tuning and S3 I/O optimization.

Preferred Skills

• Experience with large-scale data lake architectures.
• Knowledge of data governance and cataloging tools.

Key Skills

Education

Any Graduate