Description

You will build and maintain data platforms using Python, Spark, and PySpark on AWS. You are responsible for designing data pipelines, migrating workloads, and optimizing queries.

Responsibilities

• Design and implement data pipelines, migrating workloads to PySpark on AWS.
• Develop Scala/Spark jobs for data transformation, aggregation, and query optimization.
• Write unit tests for Spark transformations and helper methods.
• Document code using Scaladoc-style standards.
• Integrate data flows with SQL databases including Oracle, MySQL, Microsoft, and Postgres.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Proficiency in Python and Scala with a focus on functional programming.
• Hands-on experience with Spark, including RDD, DataFrame, MLlib, GraphX, and Streaming APIs.
• Experience with Big Data technologies and AWS environments.
• Working knowledge of HDFS, S3, Cassandra, and/or DynamoDB.
• Deep understanding of distributed systems concepts such as CAP theorem, partitioning, replication, consistency, and consensus.
• Experience building or maintaining cloud-native applications.
• Ability to work with SQL databases like Oracle and MySQL.

Preferred Skills

• Familiarity with serverless architectures using AWS Lambda.

Key Skills

Education

Any Graduate