Description

You will engineer and maintain quality assurance for distributed query engines.

Responsibilities

• Design and implement test cases for Spark, Trino, and Presto engines, focusing on execution engines, query planners, and optimizers.
• Profile and analyze query execution plans to identify bottlenecks and optimize jobs for efficiency and scalability.
• Conduct benchmarking on distributed query engines (e.g., TPC-DS, TPC-H) to evaluate performance under varying loads.
• Create and maintain synthetic data generation strategies for diverse test scenarios, including edge cases and large datasets.
• Develop and maintain automated test frameworks using PySpark, Scala, or SQL for continuous testing.

Required Skills

• 10+ years of professional experience.
• Strong expertise in Apache Spark and distributed query engines (Spark, Trino, Presto).
• Proven experience in automation testing frameworks using PySpark, Scala, or SQL.
• Hands-on experience with AWS / AWS Cloud and Big Data ecosystems.
• Proficiency in CICD pipelines and Infrastructure as Code (IAC).
• Experience with benchmarking tools (TPC-DS, TPC-H) for performance testing.
• Ability to profile and analyze query execution plans for optimization.
• Familiarity with open-source testing and monitoring tools.
• Bachelor's or Master's degree (B.E/B.Tech/M.Tech/MCA) from a recognized university.

Key Skills

Education

Any Graduate