Description

You will build and deploy modular data pipeline components within an agile environment.

Responsibilities

• Build and deploy Apache Airflow DAGs, AWS Glue jobs, and Glue crawlers using CI/CD processes.
• Translate business and functional requirements into actionable technical specifications.
• Extract, transform, and load data from diverse sources in collaboration with technology teams.
• Design efficient data storage and retrieval methods using modern data modeling techniques.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python, SQL, yaml, and PySpark.
• Hands-on experience with Apache Airflow for pipeline orchestration.
• Experience with AWS services including S3, Glue, Athena, Lake Formation, Lambda, SNS, SQS, ECS, ECR, and Fargate.
• Experience extracting data from MongoDB, DynamoDB, Snowflake, Redshift, Postgres, and RDS.
• Knowledge of Infrastructure as Code (IaC) and containerized workloads.
• Experience with modern software delivery methods including TDD, BDD, and CI/CD.
• Applied knowledge of Agile, Scrum, or DevOps environments.

Preferred Skills

• AWS Certified Developer Associate or AWS Certified Big Data Specialty.
• GitLab CI/CD experience.

Key Skills

Education

Any Graduate