Description

You will build and optimize large-scale data processing systems and pipelines using Spark and Java.

Responsibilities

• Develop, test, and deploy scalable big data solutions within the Hadoop ecosystem.
• Write optimized Java code to process large datasets efficiently.
• Design and implement both batch and real-time data processing pipelines using Spark.
• Monitor, troubleshoot, and enhance the performance of Spark jobs.
• Debug complex technical issues related to distributed computing and collaborate on system architecture.

Required Skills

• 10+ years of experience in big data development.
• Expertise in Java for writing high-performance code.
• Hands-on experience with Apache Spark, including RDD, Dataframe, Dataset, Spark SQL, and Spark Streaming.
• Strong knowledge of the Hadoop ecosystem, specifically HDFS and Hive.
• Proficiency in designing and building ETL pipelines for big data processing.
• Experience with query optimization and data manipulation using Hive or Impala.
• Proficiency with Git or similar version control systems.
• Strong understanding of Linux/Unix environments for development and deployment.
• Any Graduate degree.

Preferred Skills

• Experience with Apache Kafka.
• Exposure to DevOps practices and CI/CD pipelines.
• Knowledge of Python or Scala.

Key Skills

Education

Any Gradute