Description

You will develop and maintain big data pipelines using Scala and Apache Spark.

Responsibilities

• Design, develop, and maintain scalable data processing pipelines using Spark and Scala.
• Optimize Spark jobs for performance and efficiency.
• Integrate data from various sources, including structured, semi-structured, and unstructured data.
• Collaborate with cross-functional teams to gather requirements and deliver data-driven solutions.
• Ensure data quality and implement monitoring and logging.

Required Skills

• 8+ years of programming experience in Scala.
• Hands-on experience with Apache Spark for both batch and streaming.
• Experience with ETL processes and distributed computing.
• Familiarity with data storage solutions like HDFS, Hive, or Parquet.
• Proficiency in SQL and data modeling concepts.
• Demonstrated experience with Scala.
• Experience with ETL.
• Knowledge of Spark.

Key Skills

Education

Any Graduate