Description

You will design, build, and maintain scalable data pipelines and ETL processes within the AWS ecosystem, primarily using AWS Glue and Amazon S3.

This role is on-site.

Responsibilities

• Design and develop efficient data pipelines using AWS Glue, integrating with Lambda, EMR, Redshift, Kinesis, and Athena.
• Create and optimize AWS Glue ETL jobs using Spark, Python, or Scala to clean, transform, and enrich raw data for downstream analytics.
• Utilize AWS Glue Data Catalog and crawlers to automatically discover, catalog, and manage metadata for various data sources.
• Manage Amazon S3 data storage, including setting up storage classes, lifecycle policies, and ensuring data security.
• Implement data quality checks and validation rules within the ETL process to ensure accuracy and reliability.

Required Skills

• 6+ years of experience in data engineering or similar roles.
• Strong proficiency in Python, Scala, and Apache Spark.
• In-depth knowledge of AWS Glue and Amazon S3.
• Experience with ETL process design and implementation.
• Strong SQL skills for data manipulation and analysis.
• Understanding of data warehousing concepts and data lake architectures.
• Ability to troubleshoot technical issues and optimize pipeline performance.

Preferred Skills

• Familiarity with AWS Lambda, EMR, Redshift, Kinesis, and Athena.
• Experience with data cataloging and metadata management.

Education: Any Graduate

Key Skills

Education

Any Graduate