Description

You will design and build scalable data pipelines for batch and real-time processing within the distributed ecosystem.

Since is_remote, hybrid, and on_site are all 0, no work-mode line is included.

Responsibilities

• Design and build optimized data pipelines for batch and real-time processing.
• Assess and validate application performance and component system integration.
• Test the engineering resilience of software and automation tools.
• Develop and document technical specifications, including process flow diagrams.
• Partner with development and support teams to manage the full development life-cycle.

Required Skills

• 5+ years of software development experience, including leading engineering and scrum teams.
• 3+ years of hands-on experience with MapReduce, Hive, and Spark (Core, SQL, and PySpark).
• Expertise in writing and optimizing complex SQL, Hive, and PySpark dataframes for large-scale data processing.
• Proficiency in UNIX shell scripting.
• Programming experience using Python or Scala.
• Deep understanding of Hadoop and Spark architecture and working principles.
• Experience with distributed ecosystems.

Preferred Skills

• Knowledge of cloud platforms such as GCP or AWS.
• 2+ years of experience building solutions using Kafka streams or queues.
• Experience with NoSQL databases including HBase, Couchbase, or MongoDB.

Key Skills

Education

Not specified