Description

You will lead automation testing and architect enterprise-wide initiatives for ETL testing, data migration, and transformation processes.

This role is on-site.

Responsibilities

• Architect and lead enterprise-wide automation testing initiatives for ETL, data migration, and transformation phases.
• Debug, troubleshoot, and implement solutions for complex technical issues within big data applications.
• Design and maintain CI/CD pipelines using GitLab.
• Manage large-scale enterprise big data application deployments and testing.
• Brief technology partners, stakeholders, and senior management on the benefits and constraints of technical solutions.

Required Skills

• 6+ years of IT experience focusing on testing and automation using Java and Groovy.
• Advanced SQL expertise including Joins, Aggregation, Windowing functions, CTEs, and RDBMS schema design.
• Hands-on experience with Spark, Scala, and Java programming.
• Deep understanding of Databricks, On-Prem, Structured Streaming, Delta Lake, and Delta Live Tables.
• Knowledge of Data Lake concepts such as time travel, schema evolution, and optimization.
• Experience with AWS services including S3, Glue, and Lambda for data processing.
• Proficiency with ETL/ELT processes and tools like Data Migration Service.
• Competency with GitLab, CloudWatch, and CI/CD pipeline maintenance.

Preferred Skills

• Expertise with Great Expectations or other data quality/validation frameworks.
• AWS architecture experience.

Key Skills

Education

Any Graduate