Description

Design and implement scalable data pipelines for large-scale datasets.

Responsibilities

• Build and manage data pipelines using Spark and streaming technologies like Kafka.
• Process large volumes of data from flat files, APIs, and streaming sources.
• Orchestrate complex data workflows using Airflow or similar tools.
• Apply data modeling principles to structure large-scale datasets.
• Implement DevOps techniques within cloud environments to support data lifecycles.

Required Skills

• 8+ years of hands-on experience in Cloud environments (GCP, AWS, or Azure).
• Proficiency in Python, PySpark, or Scala.
• Strong experience with distributed computing using Spark.
• Expertise in Java or Scala for Spark development.
• Advanced SQL proficiency.
• Experience with Kafka and Spark streaming.
• Hands-on experience with workflow management using Airflow.
• Ability to work with large-scale data ingestion from APIs and streaming.

Preferred Skills

• Experience with Google Cloud Platform, specifically BigQuery and Dataproc.
• Stronger proficiency in Java over Scala.

Key Skills

Education

Any Gradute