Description

You will design and develop big data applications and distributed data processing platforms using open source technologies.

Responsibilities

• Design and develop logical and physical data models for big data platforms.
• Automate workflows using Apache Airflow and create data pipelines using Apache Hive, Spark, and Kafka.
• Lead daily standups, design reviews, and mentor junior engineers.
• Manage and prioritize backlogs using Jira and act as the primary contact for assigned business domains.
• Maintain and enhance existing systems and participate in rotational on-call support.

Required Skills

• 10+ years of experience in data engineering and related fields.
• 2+ years of recent GCP experience, specifically with Dataproc, GCS, and BigQuery.
• 5+ years of hands-on experience with Hadoop, Hive, Spark, or Airflow.
• 5+ years of experience developing data warehouse solutions and data products.
• 2+ years of experience modeling and designing schemas for data lakes or RDBMS platforms.
• Proficiency in Python, Java, or Scala.
• Experience with scripting languages such as Perl or Shell.
• Proven ability to manage large datasets at multi-TB or PB scale.
• Experience working within Scrum/Agile methodologies.
• Exposure to test-driven development and automated testing frameworks.

Preferred Skills

• Experience with Gitflow and Atlassian products including BitBucket and Confluence.
• Familiarity with CI tools such as Bamboo, Jenkins, or TFS.

Key Skills

Education

Any Graduate