Description

Build and optimize batch data pipelines to manage terabyte-scale datasets with a focus on fault tolerance and SLA adherence.

Responsibilities

• Build batch data pipelines using Hadoop, Spark, or Hive.
• Design idempotent workflows using orchestrators like Airflow, Automic, or Luigi.
• Develop data models and schemas that accommodate evolving data sources.
• Write complex SQL to analyze, profile, and optimize data in BigQuery or Spark SQL.
• Collaborate with stakeholders to translate business requirements into technical solutions.

Required Skills

• 5+ years of experience managing terabyte-scale datasets.
• Proficiency with Hadoop, Apache Spark (Scala preferred), and Apache Hive.
• Experience with cloud environments: GCP (BigQuery), AWS, or Azure.
• Expertise in building idempotent workflows using Airflow, Automic, or Luigi.
• Advanced SQL skills for data profiling and optimization.
• Strong data modeling capabilities for schema design and evolution.
• Experience with REST APIs and understanding API contracts.

Preferred Skills

• Experience building near real-time streaming pipelines using Apache Kafka, Spark Streaming, or Kafka Connect.
• Working knowledge of Apache Druid, Redis, Elasticsearch, or GraphQL.
• Exposure to Looker or Tableau for dashboard development.

Key Skills

Education

Any Graduate