Description

You will build and maintain data pipelines within a Hadoop ecosystem, owning the development, testing, and deployment of distributed data processing jobs.

This role is on-site.

Responsibilities

• Develop Scala, Spark, and PySpark jobs for data transformation and aggregation.
• Write unit tests for Spark transformations and helper methods to ensure code reliability.
• Process Parquet data and manage Hive tables using Spark-SQL and Scala APIs.
• Perform code quality reviews and act as a gatekeeper for quality checks through pair programming.
• Manage production deployments and perform post-deployment verification.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Spark, Scala, and SQL.
• Hands-on experience with Hadoop, Hive, and Impala.
• Experience with Java and Shell scripting.
• Knowledge of Kafka and Spark streaming.
• Familiarity with Oracle, Ada, and Cloudera Distribution.
• Practical experience with Git, Jenkins, and CI/CD pipelines.

Preferred Skills

• Experience with Machine Learning (ML) workflows.

Key Skills

Education

Any Graduate