Description

You will build and optimize large-scale data processing systems and pipelines using Spark and Java.

This role is on-site.

Responsibilities

• Develop, test, and deploy scalable big data solutions within the Hadoop ecosystem.
• Design and implement batch and real-time data processing pipelines using Spark.
• Write optimized Java code to process large datasets efficiently.
• Monitor, troubleshoot, and enhance the performance of Spark jobs.
• Debug complex distributed computing issues and collaborate on system architecture.

Required Skills

• 10+ years of experience in big data development.
• Expertise in Java for high-performance code.
• Hands-on experience with Apache Spark (RDD, DataFrame, Dataset, Spark SQL, Spark Streaming).
• Strong knowledge of Hadoop ecosystem, specifically HDFS and Hive.
• Proficiency in designing and building ETL pipelines for big data processing.
• Experience with query optimization and data manipulation using Hive or Impala.
• Proficiency with Git or similar version control systems.
• Strong understanding of Linux/Unix environments for development and deployment.

Preferred Skills

• Experience with Apache Kafka.
• Exposure to DevOps practices and CI/CD pipelines.
• Knowledge of Python or Scala.

Key Skills

Education

Any Gradute