Description

You will build distributed, highly parallelized big data processing pipelines for massive structured and unstructured datasets in near real-time.

Responsibilities

• Build and maintain distributed data pipelines using Spark to enrich and transform corporate data for analytics and visualization.
• Deploy and continuously deliver processes on Hadoop and other Big Data platforms.
• Automate repeatable and reliable processes to improve system efficiency.
• Collaborate with analysts, business stakeholders, and QA teams to develop analytics models and test automation solutions.
• Mentor junior engineers to ensure the production of maintainable, high-quality code.

Required Skills

• 10+ years of experience in testing and test automation using Robot Framework and Python.
• Expertise in Spark and Scala, including Spark query tuning and performance optimization.
• Deep understanding of distributed systems, including CAP theorem, partitioning, replication, consistency, and consensus.
• Extensive experience with Hadoop architecture and HDFS.
• Proficiency in SQL and working with Hive.
• Strong programming skills in Scala, Bash, and Unix scripting.
• Experience implementing CI/CD by building and maintaining Jenkins pipelines.
• Proficiency with build tools such as SBT and Gradle.
• Experience working within SAFe Agile frameworks.

Preferred Skills

• AWS Cloud experience.
• ETL Testing expertise.

Key Skills

Education

Any Graduate