Description

You design, build, and optimize scalable data pipelines using Spark, Scala, and Python in distributed environments.

This role is remote.

Responsibilities

• Develop and maintain ETL/ELT pipelines for batch and streaming data using AWS services (S3, EMR, Glue, Lambda, Athena, Redshift, Kinesis).
• Implement CI/CD pipelines for data workflows and manage infrastructure-as-code using Terraform or CloudFormation.
• Optimize Spark applications for performance, memory usage, and cost efficiency while ensuring high availability.
• Implement data quality checks, validation frameworks, and metadata management for structured, semi-structured, and unstructured data.
• Troubleshoot production issues and document technical designs and best practices.

Required Skills

• 8+ years of experience in data engineering.
• Strong proficiency in Scala and Python.
• Expertise in Apache Spark and building scalable ETL/ELT pipelines.
• Hands-on experience with AWS services: S3, EMR, Glue, Lambda, Athena, Redshift, Kinesis.
• Experience with CI/CD tools and infrastructure-as-code (Terraform or CloudFormation).
• Ability to work with large datasets in distributed environments.

Key Skills

Education

Any Gradute