Description

You will design, build, and optimize scalable batch and streaming data pipelines on AWS.

This role is hybrid.

Responsibilities

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python.
• Build event-driven workflows using AWS Lambda and manage real-time streaming solutions with Kafka, KSQL, and Apache Flink.
• Implement and enforce comprehensive data quality frameworks, including validation, profiling, monitoring, and reconciliation.
• Optimize data processing performance, scalability, reliability, and cost in cloud environments.
• Collaborate with cross-functional teams to deliver reliable, production-grade data platforms and ensure data integrity.

Required Skills

• Strong hands-on experience with Python and PySpark.
• Proven expertise in AWS Glue, Lambda, and other cloud-native data services.
• Solid experience with the Kafka ecosystem, including topics, partitions, consumer groups, and streaming patterns.
• Demonstrated experience building and supporting data quality frameworks (validation rules, reconciliation checks, profiling, anomaly detection).
• Strong understanding of distributed data processing and scalable architecture patterns.
• 10+ years of experience in data engineering.

Preferred Skills

• Experience with Apache Flink for real-time stream processing and stateful computations.
• Knowledge of KSQL or other streaming SQL engines.
• Familiarity with data lake/lakehouse architectures and table formats such as Iceberg, Delta, or Hudi.

Key Skills

Education

Any Graduate