Description

Design, build, and maintain scalable batch and streaming data pipelines on AWS. You will architect event-driven systems, optimize ETL workflows, and manage data lake architectures while ensuring data quality and security.

This role is on-site.

Responsibilities

• Architect and implement event-driven data pipelines using AWS Kinesis, MSK (Kafka), Lambda, Step Functions, SQS, and SNS.
• Develop and optimize ETL/ELT workflows using Python and PySpark, focusing on performance tuning and partitioning strategies.
• Design and manage data lake and warehouse architectures using S3, Glue Catalog, Athena, and Redshift.
• Build robust streaming solutions with checkpointing, state management, idempotency, and schema evolution.
• Implement cloud security best practices including IAM, encryption (KMS), network controls, and governance.

Required Skills

• 15+ years of professional experience in Data Engineering.
• Strong hands-on experience with Python and PySpark.
• Advanced expertise with AWS services: S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, DynamoDB, Athena, Redshift.
• Proven experience building event-driven and streaming data pipelines.
• Strong SQL skills for analytics and data transformation.
• Experience with workflow orchestration tools such as Airflow or AWS Step Functions.
• Experience with CI/CD pipelines, Git, and Infrastructure as Code (Terraform or CloudFormation).

Preferred Skills

• Strong understanding of distributed systems, Spark performance tuning, and data modeling.
• Experience with monitoring and alerting tools like CloudWatch or Datadog.

Referral Bonus: ₹50,000 INR

Key Skills

Education

Bachelor's degree