Description

You will design and develop metadata-driven Data Lake solutions to ingest historic and near real-time structured, semi-structured, and unstructured data.

Responsibilities

• Design and implement ETL and ELT pipelines for data ingestion, transformation, cataloging, and storage.
• Collaborate with Data Scientists, Architects, and business experts to clarify requirements for Data Lake components.
• Manage changes in enterprise-grade operational Data Lake environments.
• Implement security best practices using AWS IAM policies and data classification techniques.
• Enforce standards for code quality, automated testing, infrastructure-as-code, and maintainability.

Required Skills

• 10+ years of experience in data management.
• 2+ years of experience managing changes in enterprise-grade operational Data Lakes.
• Proficiency in Python, PySpark, and AWS Glue ETL jobs.
• Hands-on experience with AWS services including S3, Lambda, API Gateway, CloudFormation, DynamoDB, RDS, Glue Catalog, Glue Crawler, Athena, and QuickSight.
• Strong expertise in relational and non-relational data architecture.
• Experience writing SQL queries using AWS Athena and mapping relational DB queries to Athena.
• Experience working in Agile environments using Jira and Confluence.

Preferred Skills

• AWS Certification and experience with AWS EMR.
• Experience using Gliffy for Business Process, ERD, and AWS Architecture diagrams.
• Experience setting up AWS CloudWatch and CloudTrail for monitoring and optimization.

Key Skills

Education

Bachelor's degree