Description

You will build and manage big data pipelines within an AWS environment, migrating on-premise Hadoop clusters to AWS EMR and S3.

Responsibilities

• Migrate on-premise Hadoop clusters and HDFS storage to AWS EMR and S3.
• Configure and manage EMR clusters, spinning up and terminating instances using boto3.
• Orchestrate data flows using S3, CloudWatch, Lambda, and IAM roles.
• Develop Spark jobs to run on EMR clusters and automate data workflows.
• Implement governance and auditing using AWS CloudTrail.

Required Skills

• 5 years of experience in data engineering.
• Hands-on experience with AWS EMR, S3, EC2, Lambda, CloudWatch, SQS, SNS, and RDS.
• Proficiency in Python and the AWS Python SDK (boto3) for pipeline orchestration.
• Experience creating IAM roles and policies to secure EMR, EC2, and S3 resources.
• Strong understanding of Spark architecture and experience writing Spark jobs.
• Proficiency in SQL for intermediate-level data querying.
• Familiarity with AWS DMS for migration and disaster recovery.

Preferred Skills

• Experience with PySpark.
• Knowledge of Azure, Databricks, and Snowflake.

Key Skills

Education

Not specified