Description

Design, build, and optimize large-scale data processing applications using Spark and Scala on Hadoop/CDP clusters.

This role is on-site.

Responsibilities

• Develop and migrate Spark 2 to Spark 3 applications for high-volume data processing.
• Build and tune ETL/ELT pipelines using Spark DataFrames, Datasets, and SQL.
• Write complex HiveQL and Spark SQL queries, optimizing for performance and data governance.
• Automate job orchestration and deployments using Ansible, Control-M, or cron.
• Extract data from REST APIs and integrate it into HDFS/Hive environments.

Required Skills

• 5+ years of experience with Apache Spark and Scala.
• Strong proficiency in Hadoop ecosystem components (HDFS, Hive, CDP).
• Expertise in SQL, including window functions, CTEs, and query optimization.
• Experience with Unix shell scripting for job management and error handling.
• Proficiency in Python for API extraction and data processing.
• Knowledge of data formats like Parquet, ORC, and Avro.
• Familiarity with Kerberos authentication and HDFS encryption.

Preferred Skills

• Experience with Cloudera CDP and migration from HDP.
• Understanding of CI/CD pipelines and tools like GitHub Actions or Helios.
• Knowledge of AML or financial regulatory data domains.

Key Skills

Education

Bachelor's degree