Description

You will design, build, and optimize large-scale data processing pipelines on AWS using Python and PySpark.

This role is hybrid.

Responsibilities

• Develop and optimize batch and streaming data workflows for performance, quality, and stability.
• Build data ingestion, transformation, and modeling solutions on AWS Cloud.
• Provision and manage cloud infrastructure using Infrastructure-as-Code practices.
• Implement monitoring and dashboards to track pipeline health and troubleshoot failures.

Required Skills

• 9+ years of experience in data engineering.
• Strong proficiency in Python programming.
• Hands-on expertise with PySpark and Apache Spark for large-scale processing.
• Extensive experience with AWS Cloud services and data engineering solutions.
• Deep understanding of ETL/ELT development, data modeling, and pipeline optimization.
• Ability to build scalable, reliable data engineering solutions.

Preferred Skills

• Experience with Terraform for Infrastructure as Code.
• Experience using Grafana for monitoring and visualization.
• Knowledge of Azure Data Lake and hybrid cloud environments.

Key Skills

Education

Any Graduate