Description

You will design and manage data pipelines, focusing on efficient data ingestion, processing, orchestration, and storage.

This role is on-site.

Responsibilities

• Build and maintain data pipelines using Python and PySpark.
• Oversee data flow from various sources to destinations, ensuring quality and observability.
• Develop and test applications using Pytest or PyUnit.
• Implement infrastructure automation and manage cloud resources.
• Integrate RESTful APIs within microservices architectures.

Required Skills

• 5+ years of experience in data engineering and software development.
• Advanced proficiency in Python and PySpark.
• Hands-on experience with Flask framework.
• Experience with relational databases, specifically PostgreSQL.
• Experience with NoSQL databases, specifically MongoDB.
• Proficiency with AWS services including ECS, Lambda, Step Functions, EC2, and Boto3 SDK.
• Experience with infrastructure automation using Terraform.
• Strong understanding of back-end object-oriented programming.
• Knowledge of shell scripting languages.

Preferred Skills

• AWS Data Pipeline Management (DPL) certification.

Key Skills

Education

Any Gradute