Description

You will develop, test, deploy, and debug Spark jobs using Scala within the Hadoop platform.

Responsibilities

• Develop and optimize Spark jobs utilizing Scala and Spark Core concepts.
• Implement data processing logic using Spark SQL and RDDs.
• Integrate workflows using orchestration tools like Airflow or Oozie.
• Handle data persistence and retrieval from Hadoop environments (HDFS/Hive) and AWS S3.
• Make API calls using Scala and resolve version control conflicts in Git.

Required Skills

• 5+ years of hands-on experience developing with Spark and Big Data technologies.
• In-depth knowledge of Scala functional and OOP principles (Try, Option, Future, Traits, Case Classes).
• Proficiency in Scala programming, including understanding of Type System and Implicit/Given.
• Experience working with Hadoop environments (HDFS/Hive) and AWS S3.
• Experience with workflow orchestration tools such as Airflow or Oozie.
• Familiarity with data formats including Apache AVRO, Parquet, and JSON.
• Strong command of version control using Git, including conflict resolution and branching.
• Solid grasp of data structures and algorithms.

Preferred Skills

• Knowledge of Protocol Buffers or Geospatial data analytics.
• Experience with build tools like Gradle or SBT.

Key Skills

Education

Any Gradute