Description

Design and develop large-scale data processing applications using Spark and Scala on Hadoop/CDP clusters.

This role is on-site.

Responsibilities

• Build and optimize ETL/ELT pipelines using Spark DataFrames, Datasets, and Spark SQL.
• Tune Spark jobs for performance, focusing on partitioning, caching, broadcast joins, and shuffle optimization.
• Migrate Spark 2 applications to Spark 3 on Cloudera CDP platforms.
• Write complex HiveQL and Spark SQL queries, including window functions, CTEs, and aggregations.
• Develop bash scripts for job orchestration, error management, and HDFS operations.

Required Skills

• 5+ years of experience in Big Data Engineering.
• Strong proficiency in Scala and Python.
• Expertise with Apache Spark (DataFrames, SQL, Datasets) and Hadoop ecosystem (HDFS).
• Experience with SQL tuning and Hive table management.
• Familiarity with REST API integration, OAuth2, and JSON parsing.
• Knowledge of Unix shell scripting and job scheduling tools like Control-M or Ansible.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Cloudera CDP (7.x) and HDP migration.
• Familiarity with CI/CD pipelines (GitHub Actions) and Git.

Key Skills

Education

Bachelor's degree