Description

You will develop Spark-based solutions to clean, transform, and analyze large datasets for production pipelines and ad-hoc analysis.

Responsibilities

• Create Scala and Spark jobs for data transformation and aggregation.
• Design scalable data processing pipelines.
• Produce unit tests for Spark transformations and helper methods.
• Write Scaladoc-style documentation for all code.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Scala with a focus on functional programming.
• Expertise in Apache Spark (RDD, SQL DataFrame, MLlib, GraphX, and Streaming APIs).
• Experience with Spark query tuning and performance optimization.
• Hands-on experience with ScalaTest, JUnit, and Mockito.
• Proficiency in SQL database integration (Microsoft, Oracle, Postgres, or MySQL).
• Experience working with HDFS, S3, Cassandra, or DynamoDB.
• Deep understanding of distributed systems, including CAP theorem, partitioning, replication, and consensus.

Preferred Skills

• Experience with Embedded Cassandra.

Key Skills

Education

Any Graduate