Description

You will design and build data ingestion and transformation pipelines to support product owners and analytics consumers.

Responsibilities

• Build and deploy modular pipeline components including Apache Airflow DAGs, AWS Glue jobs, and crawlers via CI/CD.
• Translate functional requirements into actionable technical build specifications.
• Collaborate with cross-functional teams to extract, transform, and load data from diverse sources.
• Perform data analysis and onboarding activities for new data sources.
• Apply data modeling techniques to design efficient storage and retrieval methods for data consumers.

Required Skills

• 5+ years of experience in data engineering and AWS data processing.
• Proficiency in Python, SQL, YAML, and PySpark.
• Hands-on experience with Apache Airflow for pipeline orchestration.
• Experience with AWS services: S3, Glue, Athena, Lake Formation, Lambda, SNS, SQS, ECS, ECR, and Fargate.
• Experience extracting data from MongoDB, DynamoDB, Snowflake, Redshift, Postgres, and RDS.
• Knowledge of containerized workloads and Infrastructure as Code (IaC).
• Experience with modern software delivery: TDD, BDD, and CI/CD.
• Experience working in Agile, Scrum, or DevOps environments.
• Strong understanding of the full development lifecycle, including testing and versioning.

Preferred Skills

• AWS Certified Developer – Associate or AWS Certified Big Data – Specialty.
• Experience with GitLab CI/CD.

Key Skills

Education

Any Graduate