Description

This role requires you to develop and manage big data processing pipelines using Spark and Scala.

Responsibilities

• Develop and maintain data processing pipelines using Apache Spark and Scala.
• Implement data transformations and ETL processes across big data platforms.
• Work with distributed computing principles to handle large datasets.
• Apply knowledge of data serialization formats like Parquet, Avro, or ORC.

Required Skills

• 4+ years of experience in Information Technology.
• Bachelor's degree or foreign equivalent from an accredited institution (or 3 years progressive experience in lieu).
• At least 2 years of experience with Apache Spark, Scala, Spark SQL, and Starburst.
• Strong understanding of distributed computing principles and big data technologies.
• Experience with ETL processes and Data Warehouses.
• Knowledge of data serialization formats (Parquet, Avro, ORC).
• Familiarity with data processing and transformation techniques.

Preferred Skills

• Experience with data lakes.
• Understanding of Agile software development frameworks.

Key Skills

Education

Bachelor's degree