Description

You will build and maintain data pipelines, optimize large-scale processing jobs, and support data migration efforts.

Responsibilities

• Develop Hive queries using joins and partitions to process large datasets and load filtered data from source to edge node tables.
• Build Spark jobs using Scala and Python (PySpark) APIs, utilizing Spark SQL for structured data creation.
• Implement shell scripts to schedule full and incremental loads, automate metadata synchronization between GCS and GCP Hive, and manage job processing.
• Perform performance tuning for Spark applications, focusing on parallelism and memory optimization.
• Troubleshoot connectivity and performance issues for production-critical jobs and validate data between Teradata and GCP Hive.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and Scala.
• Hands-on experience with HiveQL, Hive, and HDFS.
• Experience developing Spark applications using PySpark and Scala.
• Knowledge of GCP services including Google Cloud Storage (GCS) and GCP Hive.
• Ability to develop shell scripts for automation and scheduling.
• Experience with Druid ingestion and data flattening.

Preferred Skills

• Experience with BigQuery.

Key Skills

Education

Master’s degree in Computer Science, Computer Engineering, Data Analytics,