Description

You will build and maintain data pipelines to clean, transform, and analyze large-scale raw data for production environments and ad-hoc analysis.

Responsibilities

• Create Scala and Spark jobs for data transformation and aggregation.
• Design and implement data processing pipelines.
• Develop unit tests for Spark transformations and helper methods.
• Write Scaladoc-style documentation for all code.

Required Skills

• 2 to 4 years of experience in data engineering.
• Strong proficiency in Scala with a focus on functional programming.
• Hands-on experience with Apache Spark (RDD, SQL DataFrame, MLlib, GraphX, and Streaming APIs).
• Expertise in Spark query tuning and performance optimization.
• Proficiency in SparkQL and SQL.
• Experience with testing frameworks including ScalaTest, JUnit, and Mockito.
• Experience with SQL database integration (MySQL, Microsoft, Oracle, or Postgres).
• Experience working with HDFS, S3, Cassandra, or DynamoDB.
• Deep understanding of distributed systems, including CAP theorem, partitioning, replication, consistency, and consensus.

Preferred Skills

• BTech, BE, BCA, or MCA.

Key Skills

Education

BTech, BE, BCA , MCA