Description

You will focus on building and testing scalable data processing solutions using PySpark.

Responsibilities

• Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.
• Architect end-to-end data validation systems in Hadoop environment for lineage and schema evolution.
• Lead system design for Hadoop/Hive test environments, covering YARN resource management and dynamic partitioning.
• Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments.
• Mentor juniors on PySpark testing basics and contribute to overall testing strategy discussions.

Required Skills

• 7+ years of experience in data/ETL testing with hands-on PySpark.
• Proficiency with Hadoop and Hive environments.
• Experience designing data quality systems using PySpark integrated with Hive metadata services.
• Familiarity with Zephyr, Jira, and ServiceNow integrated test management systems.
• Experience with API-driven automation within testing frameworks.
• Ability to design testing platforms and test data generators.

Key Skills

Education

Any Graduate