Description

You will design and implement data pipelines on AWS to process large volumes of data from diverse sources. You own the full lifecycle of data engineering tasks, from extraction to validation and deployment.

Responsibilities

• Design and implement data pipelines using AWS services including Glue, Lambda, Redshift, Athena, S3, EC2, and IAM.
• Develop ETL processes using Python and SQL to handle data transformation and integration.
• Perform data cleansing and validation to ensure data integrity across warehousing systems.
• Integrate CI/CD pipelines to build artifacts and deploy changes to higher environments.
• Manage data scheduling and orchestration using frameworks like Airflow or AWS Step Functions.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and SQL for data manipulation and pipeline development.
• Hands-on experience with AWS data services: Glue, Glue Crawlers, Lambda, Redshift, Athena, S3, EC2, and IAM.
• Experience setting up monitoring and logging using AWS CloudWatch and configuring alerts.
• Knowledge of scheduling frameworks such as Airflow or AWS Step Functions.
• Experience with cloud deployment and CI/CD integration.
• Strong understanding of data warehousing concepts and data validation techniques.
• Bachelor's degree or equivalent graduate education.

Key Skills

Education

ANY GRADUATE