Description

You will design and develop big data applications and manage end-to-end data pipelines within a distributed processing environment.

Responsibilities

• Design and develop big data applications using open source technologies.
• Develop logical and physical data models for big data platforms.
• Automate workflows using Apache Airflow and create pipelines using Hive, Spark, Scala, and Kafka.
• Lead daily standups, design reviews, and backlog grooming using Jira.
• Mentor junior engineers and act as the primary point of contact for assigned business domains.

Required Skills

• 8+ years of experience with Spark, Scala, Hive, and SQL.
• 8+ years of experience in ETL processes and data pipeline development.
• 5+ years of experience in GCP, including GCS, Dataproc, and BigQuery.
• 4+ years of experience with Hadoop, Airflow, or similar workflow orchestration.
• 2+ years of experience in data modeling using Erwin for data lakes or RDBMS.
• Proficiency in Python, Java, and Scala.
• Experience with scripting languages such as Perl or Shell.
• Experience managing large datasets at multi-TB or PB scale.
• Bachelor's Degree in Computer Science or equivalent experience.

Preferred Skills

• Experience with Gitflow and Atlassian products (BitBucket, Confluence).
• Familiarity with CI tools like Bamboo, Jenkins, or TFS.

Key Skills

Education

Any Graduate