Description

Design and maintain scalable data pipelines on AWS, handling batch and real-time processing with PySpark, Kafka, and Kinesis.

This role is hybrid.

Responsibilities

• Build and optimize data ingestion, transformation, and orchestration workflows using AWS Glue, Lambda, and Step Functions.
• Develop streaming pipelines for high-volume IoT and event data using Kafka or AWS Kinesis.
• Write complex SQL queries and manage data in Redshift, Athena, and DynamoDB for analytics and reporting.
• Ensure data quality, security, and performance across S3, EMR, and EC2 infrastructure.
• Collaborate with architects and product teams to deliver end-to-end data solutions.

Required Skills

• 4–10 years of data engineering experience, with at least 8+ years in relevant roles.
• Strong proficiency in Python and PySpark for large-scale data processing.
• Hands-on experience with AWS services: S3, EC2, EMR, Glue, Lambda, Step Functions, Athena, Redshift, and DynamoDB.
• Experience with streaming technologies such as Kafka or AWS Kinesis.
• Advanced SQL skills for data validation, reporting, and analytics.
• Understanding of data modeling, distributed systems, and IoT data pipelines.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

Preferred Skills

• Experience with Databricks on AWS and Apache Airflow for orchestration.
• Familiarity with cloud data warehouses like Snowflake.
• Knowledge of CI/CD, Infrastructure-as-Code, and AI/ML data pipelines.

Key Skills

Education

Bachelor's or Master's degrees