Description

You will build and maintain data pipelines within a Hadoop ecosystem, focusing on data transformation, aggregation, and quality assurance.

This role is remote.

Responsibilities

• Develop Scala, Spark, and PySpark jobs for data transformation and aggregation.
• Write unit tests for Spark transformations and helper methods.
• Use Spark-SQL to read Parquet data and create Hive tables via the Scala API.
• Perform code quality reviews and act as a gatekeeper through pair programming.
• Prepare design and operations documentation for production deployments and verification.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Hadoop, Spark, Scala, and Java.
• Experience with Hive, Impala, and Cloudera Distribution.
• Hands-on expertise with SQL, Shell Scripting, and Ada.
• Knowledge of CI/CD tools including Git and Jenkins.
• Strong understanding of data structures and distributed processing.
• Familiarity with Oracle, Kafka, Spark streaming, and ML.

Preferred Skills

• Experience with PySpark for data processing.

Key Skills

Education

Any Graduate