Description

Lead the design and development of distributed, highly parallelized data processing pipelines for massive structured and unstructured datasets.

Responsibilities

• Design and build Scala/Spark jobs for complex data transformation and aggregation.
• Develop distributed pipelines to process data in near real-time.
• Write unit tests for Spark transformations and maintain Scaladoc documentation.
• Automate repeatable processes on Hadoop and other Big Data platforms.
• Mentor junior engineers to ensure the delivery of maintainable, high-quality code.

Required Skills

• 5+ years of experience in data engineering.
• Expertise in Scala and Apache Spark (including Spark 2.x and query tuning).
• Deep understanding of Hadoop architecture and HDFS.
• Strong proficiency in SQL, Bash, and Unix/Linux environments.
• Experience with build tools such as SBT or Gradle.
• Hands-on experience implementing CI/CD using Jenkins pipelines.
• Knowledge of distributed systems concepts including CAP theorem, partitioning, and replication.
• Experience working within SAFe Agile frameworks.
• Proficiency with Git, Bitbucket, and Jira.

Preferred Skills

• AWS Cloud certification.
• Experience with Hive SQL.

Key Skills

Education

Any Graduate