Description

You will develop and maintain large-scale data processing pipelines using Scala and the Spark ecosystem.

Responsibilities

• Build and optimize data processing jobs using Apache Spark and Spark SQL.
• Implement distributed computing frameworks to handle large-scale data processing.
• Manage and maintain CI/CD pipelines using tools like Jenkins, Gradle, and Git/BitBucket.
• Collaborate with cross-functional teams within an agile development environment.
• Write and maintain Linux scripts to support data operations.

Required Skills

• 5+ years of technical experience in data engineering or related roles.
• Proficiency with Apache Spark and Scala.
• Hands-on experience with Spark SQL.
• Knowledge of big data technologies including Hadoop, HDFS, and HBase.
• Practical experience with Linux scripting.
• Experience with Git, BitBucket, Gradle, or Jenkins for CI/CD.
• Familiarity with Jira and Confluence for project tracking.
• Experience working in agile environments.
• Any Graduate degree.

Key Skills

Education

Any Graduate