Description

You will be responsible for testing ETL and batch scripts using Python and SQL.

Responsibilities

• Analyze and test data within ETL/batch processes.
• Generate detailed test reports for data quality checks.
• Query file-based systems using tools like Athena and Jupyter Notebooks.
• Test data integrity across various AWS services.

Required Skills

• 10+ years of hands-on experience with Python and Spark.
• Proficiency in SQL for testing data pipelines.
• Experience utilizing AWS services including S3 and EC2.
• Hands-on experience with Athena and Jupyter Notebooks.
• Familiarity with DataBricks.

Preferred Skills

• Exposure to Delta tables and Unity Catalog concepts.
• Knowledge of Machine learning model output testing.

Key Skills

Education

Any Graduate