Description

You will lead automation testing and architecture for enterprise-scale ETL, data migration, and transformation initiatives.

Responsibilities

• Architect and lead enterprise-wide automation testing strategies for ETL and data migration phases.
• Debug, troubleshoot, and implement solutions for complex technical issues within big data applications.
• Design and maintain GitLab CI/CD pipelines to support deployment and testing.
• Brief technology partners, stakeholders, and senior management on the benefits and constraints of proposed technical solutions.
• Optimize data processing by implementing indexing and partitioning strategies.

Required Skills

• 12+ years of IT experience with a focus on testing and automation using Java and Groovy.
• Advanced SQL expertise including joins, aggregations, windowing functions, CTEs, and RDBMS schema design.
• Hands-on experience with Databricks, Spark (Scala and Java), Structured Streaming, Delta Lake, and Delta Live Tables.
• Deep understanding of Data Lake concepts including time travel, schema evolution, and optimization.
• Practical knowledge of streaming data pipelines versus batch systems and ETL/ELT processes.
• Experience with AWS services including S3 and Lambda for data configuration and optimization.
• Proficiency with GitLab, CloudWatch, and CI/CD pipeline maintenance.
• Familiarity with Schema Registry and message formats such as Avro and ORC.
• Experience testing large-scale enterprise big data application deployments.

Preferred Skills

• Expertise with Great Expectations or similar data quality and validation frameworks.
• Experience architecting solutions within an AWS environment.

Key Skills

Education

Any Graduate