Description

You will build and optimize large-scale data pipelines and streaming architectures.

This role is on-site.

Responsibilities

• Build batch data pipelines with a focus on optimization, SLA adherence, and fault tolerance.
• Develop idempotent workflows using orchestrators like Airflow, Luigi, or Automic.
• Design data schemas and models that accommodate evolving data sources and complex joins.
• Construct near real-time streaming pipelines using Apache Kafka and Spark Streaming.
• Analyze, profile, and optimize data using SQL in BigQuery or Spark SQL.

Required Skills

• 12+ years of experience in data engineering.
• Proficiency with Apache Spark (Scala preferred) and Hadoop.
• Experience with Apache Hive and Spark SQL.
• Cloud platform experience, specifically GCP, AWS, or Azure.
• Hands-on experience with Apache Kafka, Kafka Connect, and Spark Streaming.
• Strong SQL skills for data profiling and optimization.
• Experience with REST APIs, including understanding API contracts, telemetry, and stress testing.
• Working knowledge of Apache Druid, Redis, Elasticsearch, or GraphQL.

Preferred Skills

• Specific expertise in Google Cloud Platform (GCP) and BigQuery.

Key Skills

Education

Any Gradute