Description

Build and maintain event-driven data pipelines and analytics infrastructure on AWS. You will design streaming solutions, optimize Spark-based transformations, and ensure data quality and security.

This role is on-site.

Responsibilities

• Build and maintain event-driven data pipelines using AWS Kinesis, MSK/Kafka, Lambda, Step Functions, SQS/SNS, and Glue/EMR.
• Develop ETL/ELT workflows using Python and PySpark, implementing partitioning strategies and optimizing Spark performance for scalability and cost efficiency.
• Design and manage data lake and warehouse structures using S3, Glue Catalog, Athena, and Redshift.
• Implement streaming solutions with checkpointing, stateful transformations, idempotency, and schema evolution.
• Create reusable frameworks and CI/CD pipelines for automated deployments, while mentoring junior engineers and leading design reviews.

Required Skills

• 8+ years of professional experience in Data Engineering with strong Python and PySpark expertise.
• Advanced hands-on experience with AWS services: S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, DynamoDB, Athena, and Redshift.
• Deep experience building event-driven and streaming data pipelines.
• Strong SQL experience for analytical and ETL workloads.
• Experience with workflow orchestration tools such as Airflow or Step Functions.
• Hands-on experience with CI/CD, Git, and Infrastructure-as-Code (Terraform or CloudFormation).
• Strong understanding of distributed systems, Spark performance tuning, data modeling, and cloud cost optimization.

Preferred Skills

• Experience working on Java or Spring Boot is an advantage.
• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree