Description

You will build and maintain data platforms using Python, Spark, and PySpark on AWS.

Responsibilities

• Design and implement data pipelines and handle migrations to PySpark on AWS.
• Develop Scala/Spark jobs for data transformation and aggregation.
• Write unit tests for Spark transformations and helper methods.
• Optimize Spark queries to improve performance.
• Document code using Scaladoc-style standards.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency in Python, Scala (functional programming), and Spark.
• Experience with Spark APIs including RDD, DataFrame, MLlib, GraphX, and Streaming.
• Hands-on experience with AWS, S3, and building cloud-native applications.
• Knowledge of HDFS, Cassandra, or DynamoDB.
• Experience integrating with SQL databases such as Microsoft, Oracle, Postgres, or MySQL.
• Deep understanding of distributed systems, including CAP theorem, partitioning, replication, consistency, and consensus.

Preferred Skills

• Familiarity with serverless architectures using AWS Lambda.

Key Skills

Education

Any Graduate