Description

You will design and maintain scalable data pipelines on AWS, handling batch and real-time processing for high-volume event and IoT data streams.

This role is hybrid.

Responsibilities

• Build batch and real-time data processing solutions using PySpark and Python.
• Develop streaming pipelines using Kafka or AWS Kinesis for IoT sensor data.
• Implement data ingestion, transformation, and orchestration workflows using AWS Glue, Lambda, and Step Functions.
• Write optimized SQL queries for analytics and data validation using Redshift and Athena.
• Ensure data quality, security, and performance optimization across S3, EC2, EMR, and DynamoDB.

Required Skills

• 4–10 years of experience in data engineering with strong Python and PySpark skills.
• Hands-on experience with AWS services: S3, EC2, EMR, Glue, Lambda, Step Functions, Athena, Redshift, and DynamoDB.
• Advanced SQL proficiency for large datasets and analytics.
• Experience with Kafka or AWS Kinesis for streaming architectures.
• Strong understanding of data modeling and distributed systems.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with Databricks on AWS or Apache Airflow for orchestration.
• Familiarity with CI/CD, Infrastructure-as-Code, and AI/ML data pipelines.

Key Skills

Education

Bachelor's or Master's degrees