Description

You will design and develop analytics workloads and data transformation pipelines for large-scale datasets.

Responsibilities

• Design and develop big data analytics workloads using Apache Spark and Scala.
• Create and optimize data transformation pipelines using Spark or Apache Flink.
• Migrate existing analytics workloads from cloud platforms to open-source Apache Spark infrastructure on Kubernetes.
• Lead and mentor a team of data engineers, analysts, and architects on architectural decisions and best practices.
• Collaborate with data scientists, business analysts, and developers to integrate data solutions.

Required Skills

• 5+ years of experience in data architecture or engineering.
• In-depth knowledge of Apache Spark, including Spark SQL, DataFrame APIs, Structured Streaming, and MLlib.
• Proficiency in Scala or Java.
• Hands-on experience with Kafka and Apache Flink in both streaming and batching modes.
• Expertise in performance tuning and optimizing Spark jobs.
• Strong experience with Spark production instances.
• Advanced data modeling and optimization techniques for large-scale datasets.
• Deep understanding of Data Lake, Big Data, ETL processes, and data warehousing.
• Knowledge of lakehouse technologies such as Delta Lake and Apache Iceberg.
• Experience with AWS.

Preferred Skills

• Ability to convey complex technical concepts to non-technical stakeholders.

Key Skills

Education

Any Graduate