Description

You will build and maintain scalable Spark-based data pipelines for production and ad-hoc analysis.

This role is hybrid.

Responsibilities

• Develop Scala and Spark jobs for data transformation, aggregation, and cleaning.
• Design and implement scalable data processing pipelines.
• Write unit tests for Spark transformations and helper methods using ScalaTest, JUnit, and Mockito.
• Produce Scaladoc-style documentation for all code.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Scala with a focus on functional programming.
• Expertise in Apache Spark (RDD, SQL DataFrame, MLlib, GraphX, and Streaming APIs).
• Experience with Spark query tuning and performance optimization.
• Proficiency in SQL database integration (Microsoft, Oracle, Postgres, or MySQL).
• Experience working with HDFS, S3, Cassandra, or DynamoDB.
• Deep understanding of distributed systems, including CAP theorem, partitioning, replication, and consensus.

Preferred Skills

• Experience with Embedded Cassandra.

Key Skills

Education

Any Graduate