Description

You will design, develop, and optimize large-scale data pipelines and ETL workflows within an AWS environment.

Responsibilities

• Build and optimize scalable ETL workflows using PySpark and Apache Spark.
• Leverage AWS services including EMR, S3, Lambda, and Glue to manage data pipelines.
• Orchestrate and schedule complex workflows using Apache Airflow.
• Integrate data from diverse sources while ensuring high quality, consistency, and reliability.
• Write efficient SQL queries for data processing, validation, and troubleshooting production issues.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Python and PySpark.
• Hands-on experience with Apache Spark and SQL.
• Deep knowledge of AWS services: EMR, S3, Lambda, and Glue.
• Experience orchestrating pipelines with Apache Airflow.
• Proven ability to work with large datasets and optimize Spark jobs.
• Experience building data lakes and data warehouses on AWS.
• Working knowledge of DynamoDB.
• Any Graduate degree.

Preferred Skills

• Experience with Kafka or other real-time streaming platforms.
• Familiarity with DevOps tools like Terraform or CloudFormation.
• Exposure to NoSQL databases such as MongoDB.

Key Skills

Education

Any Graduate