Description

Design and maintain scalable batch and streaming data pipelines on AWS.

This role is on-site.

Responsibilities

• Architect event-driven pipelines using AWS Kinesis, MSK, Lambda, Step Functions, SQS, and SNS.
• Optimize ETL/ELT workflows and Spark performance tuning with Python and PySpark.
• Build and manage data lake/warehouse architectures using S3, Glue Catalog, Athena, and Redshift.
• Implement streaming solutions with checkpointing, state management, and schema evolution.
• Create reusable frameworks, internal libraries, and CI/CD pipelines for automated deployments.

Required Skills

• 15+ years of professional experience in Data Engineering.
• Advanced expertise with Python and PySpark.
• Strong hands-on experience with AWS services: S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, DynamoDB, Athena, Redshift.
• Proven experience building event-driven and streaming data pipelines.
• Strong SQL skills for analytics and data transformation.
• Experience with workflow orchestration tools such as Airflow or AWS Step Functions.
• Experience with CI/CD pipelines, Git, and Infrastructure as Code (Terraform or CloudFormation).
• Strong understanding of distributed systems, Spark performance tuning, and data modeling.

Preferred Skills

• Experience with data quality, monitoring, observability, and alerting tools like CloudWatch and Datadog.
• Knowledge of cloud security best practices including IAM, encryption (KMS), and network controls.
• Ability to lead technical design reviews and mentor junior engineers.

Key Skills

Education

Bachelor's degree