Description

Lead enterprise-wide automation testing for ETL, data migration, and transformation processes. Architect frameworks and validate streaming data pipelines.

This role is on-site.

Responsibilities

• Architect automation testing frameworks for ETL testing across multiple phases.
• Design and maintain CI/CD pipelines using GitLab.
• Validate streaming data pipelines, differentiating between batch and streaming systems.
• Optimize AWS Lambda configurations and S3 data processing.

Required Skills

• 8+ years of IT experience with a focus on testing and automation.
• Strong programming skills in Java, Groovy, and Spark Scala.
• Hands-on experience with Databricks, Structured Streaming, Delta Lake, and Delta Live Tables.
• Deep understanding of Data Lake concepts including time travel, schema evolution, and optimization.
• Expertise in ETL/ELT processes and tools like Data Migration Service.
• Proficiency with GitLab and CloudWatch for CI/CD support.
• Experience with AWS services including S3 and Lambda.
• Bachelor's Degree.

Preferred Skills

• Familiarity with Great Expectations or similar data quality frameworks.
• Architecture experience within an AWS environment.

Key Skills

Education

Bachelor's Degree