Description

Key Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines on AWS
• Architect and implement event-driven data pipelines using services such as:
  • AWS Kinesis / MSK (Kafka)
  • Lambda
  • Step Functions
  • SQS / SNS
  • Glue and EMR
• Develop and optimize ETL / ELT workflows using Python and PySpark
• Implement Spark-based transformations, performance tuning, and partitioning strategies
• Design and manage data lake and warehouse architectures using S3, Glue Catalog, Athena, and Redshift
• Build robust streaming solutions with checkpointing, state management, idempotency, and schema evolution
• Ensure strong data quality, monitoring, observability, and alerting (CloudWatch, Datadog, etc.)
• Implement cloud security best practices including IAM, encryption (KMS), network controls, and governance
• Create reusable frameworks, internal libraries, and CI/CD pipelines for automated deployments
• Collaborate with data scientists, analysts, and business stakeholders to deliver trusted datasets
• Lead technical design reviews, mentor junior engineers, and set engineering standards

Required Qualifications (Must-Have)

• 15+ years of professional experience in Data Engineering
• Strong hands-on experience with Python and PySpark
• Advanced expertise with AWS services, including:
  • S3, Glue, EMR
  • Lambda, Step Functions
  • Kinesis/MSK, DynamoDB
  • Athena, Redshift
• Proven experience building event-driven and streaming data pipelines
• Strong SQL skills for analytics and data transformation
• Experience with workflow orchestration tools such as Airflow or AWS Step Functions
• Experience with CI/CD pipelines, Git, and Infrastructure as Code (Terraform or CloudFormation)
• Strong understanding of:
  • Distributed systems
  • Spark performance tuning
  • Data modeling
  • Cloud cost optimization
• Solid knowledge of data security, encryption, networking, and cloud compliance

Key Skills

Education

Bachelor's degree