Description

You will build and maintain data platforms using Python, Spark, and PySpark on AWS.

Responsibilities

• Design and implement data pipelines and migrate workloads to PySpark on AWS.
• Develop Scala/Spark jobs for data transformation and aggregation.
• Optimize Spark queries to improve performance.
• Write unit tests for Spark transformations and helper methods.
• Document code using Scaladoc-style standards.

Required Skills

• Proficiency in Python and Scala, specifically functional programming.
• Extensive experience with Spark and PySpark.
• Knowledge of Spark APIs including RDD, DataFrame, MLlib, GraphX, and Streaming.
• Experience with HDFS, S3, Cassandra, and/or DynamoDB.
• Deep understanding of distributed systems, including CAP theorem, partitioning, replication, consistency, and consensus.
• Experience building or maintaining cloud-native applications.
• Ability to integrate with SQL databases such as Microsoft, Oracle, Postgres, or MySQL.

Preferred Skills

• Familiarity with serverless approaches using AWS Lambda.

Key Skills

Education

Not specified