Description

You will build, optimize, and migrate large-scale data pipelines on AWS using Python and Spark.

This role is remote.

Responsibilities

• Build and maintain data pipelines using Spark to support large-scale platform features.
• Develop libraries to standardize data processing workflows.
• Migrate data platforms into AWS environments and implement automation for data processes.
• Partner with architects and analysts to design technical architectures for enterprise projects.
• Advise and coach other data professionals on engineering standards and practices.

Required Skills

• 5+ years of experience in a data engineering role.
• Proficiency in Python and SQL.
• Strong experience building data pipelines with Spark.
• Experience building data processing infrastructure within AWS (e.g., EMR or Glue).
• Experience with relational datastores, NoSQL datastores, and cloud object stores.
• Legal right to work in the USA.

Preferred Skills

• Production experience with ACID compliant formats such as Hudi, Iceberg, or Delta Lake.
• Experience with Infrastructure as Code, specifically Terraform.

Key Skills

Education

Any Graduate