Description

Lead technical setup and proof of concept for new Spark-based implementations, including cluster configuration, performance tuning, and integration best practices.

This role is remote.

Responsibilities

• Build initial proof of concepts for planned Spark use cases.
• Configure and set up Spark clusters from scratch.
• Establish engineering best practices for integration and performance.
• Manage log aggregation, management, and processing at scale.
• Write shell scripts and Python glue code as required.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expertise in Spark cluster configuration and performance tuning.
• Proficiency with PySpark and Spark-based ML.
• Experience with Presto configuration and performance.
• Strong Python skills and experience using Python notebooks.
• Hands-on experience with Oracle database performance.
• Knowledge of S3, Spark, and Presto integration.
• Familiarity with Hadoop infrastructure.
• Competency in DevOps automation using Python, Ansible, or Terraform.
• Ability to work with Kubernetes and Kafka/Pulsar.

Preferred Skills

• Experience leading initial technical setups and POCs.

Key Skills

Education

Any Graduate