Description

Lead and architect enterprise-wide automation testing initiatives for ETL, data migration, and transformation processes.

Responsibilities

• Architect automation testing frameworks for ETL testing across multiple testing phases.
• Design and maintain CI/CD pipelines using GitLab.
• Validate streaming data pipelines, differentiating between batch and streaming systems.
• Manage data quality and validation using specialized frameworks.
• Optimize AWS Lambda configurations and S3 data processing.

Required Skills

• 8+ years of IT experience with a focus on testing and automation.
• Strong programming skills in Java, Groovy, and Spark Scala.
• Hands-on experience with Databricks, Structured Streaming, Delta Lake, and Delta Live Tables.
• Deep understanding of Data Lake concepts including time travel, schema evolution, and optimization.
• Expertise in ETL/ELT processes and tools like Data Migration Service.
• Proficiency with GitLab and CloudWatch for CI/CD support.
• Experience with AWS services including S3 and Lambda.
• Knowledge of Schema Registry and message formats such as Avro and ORC.
• Bachelor's Degree.

Preferred Skills

• Familiarity with Great Expectations or similar data quality frameworks.
• Architecture experience within an AWS environment.
• Experience using AWS tools to view and process massive data at scale.

Key Skills

Education

Bachelor's Degree