Description

Design and develop large-scale analytics workloads and data transformation pipelines.

This role is on-site.

Responsibilities

• Design and develop analytics workloads using Apache Spark and Scala for big data processing.
• Create and optimize data transformation pipelines using Spark or Apache Flink.
• Migrate existing analytics workloads from cloud platforms to open-source Apache Spark infrastructure running on Kubernetes.
• Lead and mentor a team of data engineers, analysts, and architects on architectural decisions and best practices.
• Collaborate with data scientists, business analysts, and developers to integrate data solutions.

Required Skills

• 5+ years of experience with production-scale Spark instances.
• In-depth knowledge of Apache Spark APIs, including Spark SQL, DataFrame APIs, Structured Streaming, and MLlib.
• Proficiency in Scala or Java.
• Experience with Kafka and Apache Flink, including streaming and batching modes.
• Expertise in data modeling and optimization for large-scale datasets.
• Strong understanding of Data Lake, Big Data, ETL processes, and data warehousing.
• Knowledge of lakehouse storage technologies such as Delta Lake and Apache Iceberg.
• Experience with AWS.
• Ability to communicate complex technical concepts to non-technical stakeholders.

Key Skills

Education

Any Graduate