Description

You will build and maintain scalable data processing pipelines using Scala and Apache Spark.

This role is on-site.

Responsibilities

• Develop, test, and maintain scalable data processing pipelines using Scala and Apache Spark.
• Implement data transformation and ETL workflows to handle structured and unstructured data.
• Utilize Python for data processing, scripting, and integrating Spark workflows.
• Optimize performance of Spark applications through tuning, partitioning, and caching strategies.
• Participate in code reviews, design discussions, and ensure adherence to best practices.

Required Skills

• Strong experience in Scala and Apache Spark (Spark Core, Spark SQL, DataFrames, RDDs).
• Proficiency in Python for data manipulation, scripting, and automation.
• 8+ years of proven experience with distributed computing concepts and large-scale data processing.
• Knowledge of ETL development, data warehousing, and data modeling techniques.
• Familiarity with Big Data ecosystems such as Hadoop, Hive, Kafka, or HBase.
• Excellent communication and collaboration skills.

Preferred Skills

• Experience with cloud platforms such as AWS, GCP, or Azure (particularly with Spark services).
• Familiarity with streaming data processing using Spark Streaming or Structured Streaming.
• Knowledge of data governance, security, and compliance practices.

Key Skills

Education

Any Gradute