Description

You will design and develop big data applications and distributed data processing platforms using open source technologies.

Responsibilities

• Design and develop logical and physical data models for big data platforms.
• Automate workflows using Apache Airflow and create data pipelines with Apache Hive, Apache Spark, and Apache Kafka.
• Lead daily standups, design reviews, and mentor junior engineers.
• Maintain and enhance existing systems while participating in rotational on-call support.
• Groom and prioritize backlogs using JIRA and act as the point of contact for assigned business domains.

Required Skills

• 2+ years of recent experience with GCP, specifically Dataproc, GCS, and BigQuery.
• 5+ years of hands-on experience developing data warehouse solutions and data products.
• 5+ years of experience with Hadoop, Hive, or Spark, and Airflow or similar orchestration tools.
• 2+ years of experience modeling and designing schemas for data lakes or RDBMS platforms.
• Proficiency in Python, Java, or Scala.
• Experience with scripting languages such as Perl or Shell.
• Proven experience managing large datasets at multi-TB or PB scale.
• Experience with test-driven development and automated testing frameworks.
• Background working in Scrum/Agile development methodologies.

Preferred Skills

• Experience with Gitflow and Atlassian products including BitBucket, JIRA, and Confluence.
• Experience with CI tools such as Bamboo, Jenkins, or TFS.

Key Skills

Education

Any Graduate