Description

Build and maintain scalable data pipelines for Media Measurement using PySpark, AWS EMR, and Airflow.

This role is on-site.

Responsibilities

• Design, develop, test, and deploy big data applications and ETL pipelines with a focus on scalability and efficiency.
• Take ownership of technical implementations and system architecture within a distributed, cross-functional agile team.
• Collaborate with data producers and consumers to align on data exchange formats and interfaces for seamless integration.
• Recommend improvements to data reliability and quality across all processing pipelines.
• Explore and integrate AI-based workflows to enhance engineering automation.

Required Skills

• 6+ years of relevant experience in Data Engineering.
• Hands-on expertise with PySpark, Spark, and AWS EMR.
• Proficiency in Python, Java, or Scala.
• Strong SQL skills and experience with Data Warehousing and ETL processes.
• Experience building and managing workflows with Airflow.
• Understanding of AWS cloud services.

Preferred Skills

• Experience with Terraform, CI/CD, and infrastructure as code.
• Familiarity with Data Lake, Lake Formation, and Apache Iceberg.
• Knowledge of testing, clean code practices, and pair programming.

Key Skills

Education

Any Graduate