Description

You will design and implement scalable data lake architectures and automated ETL pipelines within an AWS environment.

Responsibilities

• Build and optimize ETL pipelines using AWS Glue, EMR, and Spark.
• Design scalable data lake architecture using AWS S3 and Lake Formation.
• Implement event-driven workflows utilizing EventBridge, SNS, and SQS.
• Automate infrastructure provisioning with Terraform and CloudFormation.
• Expose APIs for data subscription and sharing via API Gateway and AWS Lambda.

Required Skills

• 12+ years of experience in data engineering roles.
• Proficiency in Python, SQL, and Apache Spark for distributed data processing.
• Hands-on experience with AWS S3, Glue, EMR, and Athena.
• Experience implementing event-driven architectures using SNS, SQS, and EventBridge.
• Practical knowledge of Terraform or CloudFormation for IaC.
• Experience with Amazon Neptune DB for graph-based metadata management.
• Strong understanding of data lake architecture, governance, and security.
• Ability to implement IAM policies and access controls for data compliance.
• Experience developing self-serve portals using Angular and backend integration.

Preferred Skills

• Familiarity with AWS Lambda for serverless data processing.

Key Skills

Education

Any Gradute