Description

You will build and maintain data pipelines handling large volumes of data from flat files, APIs, and streaming sources.

This role is on-site.

Responsibilities

• Build distributed computing processes using Spark with Java or Scala.
• Manage data workflows using Airflow or similar orchestration tools.
• Develop streaming data pipelines using Kafka and Spark Streaming.
• Implement data modeling patterns to support data architecture.
• Process large-scale datasets across various ingestion methods.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Spark using Java or Scala (Java preferred).
• Advanced SQL skills.
• Hands-on experience with Kafka and Spark Streaming.
• Experience with workflow management tools like Airflow.
• Proven ability to handle large data volumes from APIs and flat files.
• Strong understanding of data modeling.

Preferred Skills

• Experience with Google Cloud Platform (GCP).
• Proficiency with BigQuery and Dataproc.

Key Skills

Education

Any Gradute