Description

Design and implement scalable data pipelines on AWS to process large volumes of data from diverse sources. Own the full lifecycle from ingestion to deployment, ensuring data quality and system reliability.

Responsibilities

• Design and implement scalable data pipelines on AWS.
• Perform data cleansing and validation to ensure high data quality.
• Develop ETL processes using Python and SQL.
• Integrate CI/CD pipelines to build artifacts and deploy changes to higher environments.
• Manage scheduling and orchestration using Airflow or AWS Step Functions.

Required Skills

• 5+ years of experience in data engineering.
• Hands-on experience with AWS Glue, Glue Crawlers, and Lambda.
• Proficiency in SQL and Python for ETL development.
• Experience with AWS Redshift, Athena, and S3.
• Working knowledge of EC2 and IAM.
• Experience with monitoring and logging via AWS CloudWatch and alert setup.
• Practical knowledge of cloud deployment and CI/CD integration.
• Experience with Airflow or AWS Step Functions for orchestration.

Preferred Skills

• Experience with Apache Kafka.

Key Skills

Education

Any Graduate