Description

You will design and build distributed data processing platforms and big data applications.

Responsibilities

• Design and develop big data applications using open-source technologies.
• Create logical and physical data models for data lakes and RDBMS.
• Automate workflows with Apache Airflow and build pipelines using Hive, Spark, Scala, and Kafka.
• Lead design reviews, groom backlogs in JIRA, and participate in daily standups.
• Mentor junior engineers and maintain existing systems with rotational on-call support.

Required Skills

• 8+ years in ETL, data pipelining, SQL, Hive, Scala, and Spark.
• 4+ years with GCP, GCS, Dataproc, and BigQuery.
• 4+ years developing distributed data processing platforms with Hadoop, Hive, Scala, or Airflow.
• 2+ years in schema design and modeling (Erwin) for large-scale data.
• Proficiency in Python, Java, and Scala.
• Experience with Perl and Shell scripting.
• Proven management of multi-TB or PB-scale datasets.
• Background in Scrum/Agile and Test Driven Development.
• Bachelor's Degree in Computer Science or equivalent experience.

Preferred Skills

• Experience with Gitflow and Atlassian products (BitBucket, JIRA, Confluence).
• Familiarity with CI tools such as Bamboo, Jenkins, or TFS.
• Retail domain experience.

Key Skills

Education

Any Graduate