Description

Design, develop, and implement optimized data pipelines for batch and real-time processing. Lead engineering teams and scrum processes while documenting technical specifications and analyzing system flows.

Responsibilities

• Design and build optimized data pipelines for batch and real-time processing.
• Analyze, design, test, and implement system applications.
• Develop and document technical and functional specifications.
• Lead teams of engineers and scrum teams.
• Analyze software and system processing flows.

Required Skills

• 7+ years of software development experience.
• 3+ years of hands-on experience with Map-Reduce, Hive, and Spark (Core, SQL, and PySpark).
• Expertise in Hadoop and Spark architecture and working principles.
• Proficiency in Core Java, Python, or Scala.
• Experience writing and optimizing complex SQL (Hive/PySpark-dataframes) for large-scale data processing.
• Strong understanding of distributed ecosystems and data warehousing concepts.
• Hands-on experience with UNIX shell scripting.
• Bachelor's degree in Engineering, Computer Science, or equivalent.

Preferred Skills

• Experience designing and building solutions using Kafka streams or queues.
• Knowledge of cloud platforms (GCP/AWS) and building microservices.

Key Skills

Education

Any Gradutate