Description

Build and maintain large-scale data pipelines and machine learning models using Apache Spark.

Responsibilities

• Build and maintain data pipelines and machine learning models using Apache Spark.
• Process large-scale datasets, specifically at the petabyte scale.
• Perform performance tuning on Spark applications and regression models.
• Troubleshoot and debug Spark application failures and long-running processes.
• Optimize Spark code for scalability, performance, and maintainability.

Required Skills

• 3-4 years of proficiency with Apache Spark.
• 2-3 years of experience with AWS EMR, including Docker-compatible EMR.
• Experience using compute-intensive, memory-intensive, and GPU instances.
• 3-5 years of experience with SQL queries and databases, preferably Snowflake.
• Deep expertise in Python and PySpark.
• Proficiency in Scala or R coding.
• Experience analyzing big data and collaborating with data scientists.
• Proficiency with GitHub for codebase management.
• 14+ years of total professional experience.

Preferred Skills

• Experience with the R language.

Key Skills

Education

ANY GRADUATE