Description

You will design, build, and maintain scalable, fault-tolerant real-time and batch data pipelines to process large volumes of data.

Responsibilities

• Build and maintain data pipelines that transform raw data into governed, high-quality datasets for data scientists and analysts.
• Develop an orchestration layer for complex data transformations to ensure optimal performance and SLA adherence.
• Implement data quality checks, validation, and cleaning processes to ensure accuracy across all ingested data.
• Collaborate with cross-functional teams to gather requirements, write code, conduct unit testing, and troubleshoot issues.
• Identify automation and big data analytics solutions to address complex business problems.

Required Skills

• 4+ years of experience developing big data technologies and data pipelines.
• Proven experience managing and manipulating datasets at the terabyte (TB) scale.
• Proficiency with Apache Spark (Scala preferred) and Hadoop.
• Experience building idempotent workflows using orchestrators such as Airflow, Luigi, or Automic.
• Strong SQL skills for data analysis, profiling, and optimization, specifically in BigQuery or Spark SQL.
• Experience with cloud platforms, preferably Google Cloud Platform (GCP), AWS, or Azure.
• Competency in Python and Hive.

Preferred Skills

• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree