Description

You will design and develop big data applications and manage data pipelines at multi-TB/PB scale.

Responsibilities

• Design and develop big data applications using open source technologies.
• Develop logical and physical data models for big data platforms.
• Automate workflows using Apache Airflow and create pipelines using Hive, Spark, and Kafka.
• Mentor junior engineers, lead daily standups, and conduct design reviews.
• Maintain existing systems and participate in rotational on-call support.

Required Skills

• 5+ years of experience developing data warehouse solutions and distributed data processing platforms using Hadoop, Hive, Spark, or Airflow.
• 2+ years of recent experience building data pipelines within GCP, specifically using Dataproc, GCS, and BigQuery.
• 2+ years of experience modeling and designing schemas for data lakes or RDBMS platforms.
• Proficiency in Python, Java, or Scala.
• Experience with scripting languages including Perl and Shell.
• Experience managing large data sets at multi-TB or PB scale.
• Experience with test-driven development and automated testing frameworks.
• Background working in Scrum/Agile methodologies.
• Bachelor's Degree in Computer Science or equivalent experience.

Preferred Skills

• Experience with Gitflow and Atlassian products (BitBucket, JIRA, Confluence).
• Experience with CI tools such as Bamboo, Jenkins, or TFS.

Key Skills

Education

Any Graduate