Description

Lead engineering and scrum teams in the design, development, and implementation of large-scale data systems.

Responsibilities

• Design and develop optimized data pipelines for batch and real-time processing.
• Architect and implement solutions using Hadoop and Spark frameworks.
• Write and optimize complex SQL, Hive, and PySpark dataframes for massive datasets.
• Analyze, design, test, and document technical and functional system specifications.
• Lead software and system processing flow analysis.

Required Skills

• 10+ years of software development experience with a focus on leading engineering teams.
• 5+ years of hands-on experience with Spark (Core, SQL, PySpark), Map-Reduce, and Hive.
• 5+ years of experience in Data Engineering or Data Warehousing using Big Data technologies.
• Expertise in distributed ecosystems and Hadoop/Spark architecture.
• Proficiency in Core Java, Python, or Scala.
• 1+ years of experience building solutions with Kafka streams or queues.
• Experience with UNIX shell scripting.
• Experience leveraging GitHub, Bitbucket, and CI/CD pipelines.

Preferred Skills

• Knowledge of cloud platforms including GCP or AWS.
• Experience with NoSQL databases such as MongoDB, HBase, or Couchbase.

Key Skills

Education

Any Graduate