Description

You will design and implement data pipelines to migrate and transform data from relational databases to S3.

Responsibilities

• Write Glue ETL jobs to convert data from AWS RDS (SQL Server and Oracle) into Parquet format in S3.
• Execute and manage Glue Crawlers to catalog S3 files and maintain the AWS Glue Data Catalog.
• Develop complex SQL queries in Amazon Athena and create views for business use cases.
• Define data lifecycle management and partitioning strategies for S3 to optimize storage and querying.
• Secure data using IAM policies, S3 bucket policies, and KMS encryption.

Required Skills

• 5+ years of experience as an AWS Data Engineer.
• Proficiency in AWS Glue (ETL, Catalog, Crawlers, and Glue Studio).
• Experience with PySpark for developing and debugging Glue ETL jobs.
• Hands-on experience with AWS RDS, specifically SQL Server and Oracle DB.
• Deep knowledge of AWS S3 architecture, partitioning, and Parquet file formats.
• Strong ability to write and optimize SQL queries in Amazon Athena.
• Understanding of data security via IAM, bucket policies, and KMS.
• Knowledge of regulatory requirements like GDPR for secure data handling.
• Ability to communicate complex technical concepts to non-technical audiences.
• Any Graduate degree.

Key Skills

Education

Any Graduate