Description

You design and implement scalable data pipelines on AWS to process large volumes of data from diverse sources.

Responsibilities

• Build and maintain data pipelines using AWS Glue, Lambda, Redshift, Athena, S3, EC2, and IAM.
• Develop ETL processes using Python and SQL for data transformation and integration.
• Perform data cleansing and validation to ensure integrity across warehousing systems.
• Implement CI/CD pipelines to build artifacts and deploy changes to higher environments.
• Manage data scheduling and orchestration using Airflow or AWS Step Functions.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and SQL for data manipulation and pipeline development.
• Hands-on experience with AWS data services: Glue, Glue Crawlers, Lambda, Redshift, Athena, S3, EC2, and IAM.
• Experience setting up monitoring and logging using AWS CloudWatch and configuring alerts.
• Knowledge of scheduling frameworks such as Airflow or AWS Step Functions.
• Experience with cloud deployment and CI/CD integration.
• Strong understanding of data warehousing concepts and data validation techniques.

Preferred Skills

• Bachelor's degree or equivalent graduate education.

Key Skills

Education

ANY GRADUATE