Description

Design, build, and optimize scalable data platforms and pipelines for large-scale distributed systems.

This role is hybrid.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines.
• Perform large-scale data processing using PySpark and Spark SQL.
• Build and manage data solutions on AWS (EMR, Glue, Lambda, S3).
• Develop and optimize data warehouses using Redshift, RDS, and MySQL.
• Monitor, log, and troubleshoot pipelines using AWS CloudWatch and secure credentials with Secrets Manager.

Required Skills

• 10–12 years of experience as a Data Engineer or similar role.
• Strong proficiency in Python and SQL.
• Extensive hands-on experience with PySpark and Spark SQL.
• Deep experience with AWS Data Services: EMR, Glue, Lambda, S3, RDS, MySQL, Redshift.
• Strong understanding of distributed computing architectures.
• Proven expertise in data modeling, schema handling, and performance tuning.
• Excellent debugging and problem-solving skills.

Preferred Skills

• Experience with streaming technologies (Kafka, Kinesis, etc.).
• Exposure to CI/CD pipelines for data platforms.
• Knowledge of data governance, security, and compliance best practices.

Key Skills

Education

Any Graduate