Description

You will lead the migration of Big Data workloads from Cascading and MapReduce to Spark 3.

Responsibilities

• Spearhead the full migration lifecycle of Big Data workloads to Spark 3.
• Develop efficient, scalable Spark applications using Scala within the IntelliJ IDE.
• Design and architect Kubernetes environments for Spark 3 deployments, focusing on scalability and fault tolerance.
• Conduct Scala code reviews to enforce standards and identify performance bottlenecks.
• Optimize Spark application performance through configuration tuning, data partitioning, and caching.
• Document design specifications, migration plans, and operational procedures.

Required Skills

• 5+ years of experience in software engineering.
• Proven experience migrating Big Data workloads from Cascading and MapReduce to Spark 3.
• Expert-level proficiency in Scala programming.
• Deep understanding of Spark architecture and internals.
• Hands-on experience with IntelliJ IDE for development and debugging.
• Experience designing and managing Kubernetes environments for Spark.
• Strong knowledge of distributed computing principles and data processing paradigms.
• Proficiency in version control, testing, and CI/CD practices.
• Any Graduate degree.

Preferred Skills

• Experience with Hadoop, Hive, or Kafka.
• Familiarity with AWS, Azure, or GCP.
• Knowledge of ETL processes and data warehousing.

Key Skills

Education

Any Graduate