Description

You will build and manage scalable data pipelines using Java and Spark within a Hadoop ecosystem.

Responsibilities

• Develop and maintain scalable, reliable data pipelines to process information from multiple sources.
• Transform and aggregate data while ensuring smooth end-to-end processing.
• Optimize Spark applications using memory caching and broadcast hints for SQL queries.
• Implement deployment pipelines and manage code using Git Bash.
• Collaborate within a Scrum framework and communicate technical requirements to clients via email and phone.

Required Skills

• 5+ years of experience in Data Engineering.
• Proficiency in Java and its ecosystem.
• Hands-on experience with Spark Core, Spark SQL, RDD, Data Set, and Data Frames.
• Deep understanding of Spark and Hadoop architecture.
• Experience working with HDFS.
• Knowledge of Azure/Cloud DevOps and CI/CD pipeline concepts.
• Practical experience with Git Bash.
• Experience working in Scrum methodologies and participating in agile ceremonies.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate