Description

You will build and maintain ELT pipelines using Apache Airflow and Snowflake within an AWS cloud environment.

Responsibilities

• Design, schedule, and monitor complex data flows using Apache Airflow on AWS.
• Develop and maintain modular Python code for ETL automation and REST API integration.
• Implement and manage data storage solutions, primarily utilizing AWS S3 and Redshift.
• Monitor pipeline operations and set up failure alerts using AWS CloudWatch.

Required Skills

• 10+ years of professional experience in data engineering.
• Expert proficiency in SQL, including complex joins, window functions, and CTEs, with performance tuning skills for Snowflake.
• Proficiency in Python, specifically for scripting, automation, and data manipulation using Pandas and NumPy.
• Strong experience with Apache Airflow for workflow orchestration on AWS.
• Solid working knowledge of core AWS services (S3, IAM, CloudWatch, Lambda).
• Experience integrating and processing data from various REST APIs.
• Demonstrated understanding of data lakes versus data warehouses architecture.
• Ability to deploy container-based services in AWS with monitoring setup.

Key Skills

Education

Any Graduate