Description

You will design, build, and maintain scalable data pipelines for real-time and batch processing.

This role is on-site.

Responsibilities

• Build and maintain data pipelines that transform raw data into governed, high-quality datasets.
• Develop an orchestration layer for complex data transformations to ensure optimal performance and SLA adherence.
• Implement data quality checks, validation, and cleaning processes to ensure accuracy across ingested data.
• Collaborate with cross-functional teams to gather requirements, write code, conduct unit testing, and troubleshoot issues.

Required Skills

• 4+ years of experience developing big data technologies and data pipelines.
• Proficiency with Apache Spark (Scala preferred) and Hadoop.
• Strong SQL skills for data analysis, profiling, and optimization, specifically in BigQuery or Spark SQL.
• Experience building idempotent workflows using orchestrators such as Airflow, Luigi, or Automic.
• Experience with cloud platforms, preferably Google Cloud Platform (GCP), AWS, or Azure.
• Competency in Python and Hive.
• Proven experience managing and manipulating datasets at the terabyte (TB) scale.

Key Skills

Education

Bachelor's degree