Description

You will lead automation testing and architect enterprise-wide initiatives for ETL, data migration, and transformation processes.

Responsibilities

• Architect and lead enterprise-wide automation testing for ETL, data migration, and transformation phases.
• Debug, troubleshoot, and implement solutions for complex technical issues within big data applications.
• Design and maintain GitLab CI/CD pipelines to support deployment and testing.
• Test and optimize large-scale enterprise big data application deployments.
• Brief technology partners and stakeholders on the benefits and constraints of various technical solutions.

Required Skills

• 8+ years of IT experience with a focus on testing and automation.
• Strong programming proficiency in Java, Groovy, and Spark Scala.
• Deep expertise in Databricks, Structured Streaming, Delta Lake, and Delta Live Tables.
• Advanced SQL skills including joins, aggregations, windowing functions, CTEs, and RDBMS schema design.
• Hands-on experience with AWS services including S3, Glue, and Lambda.
• Practical knowledge of Data Lake concepts like time travel, schema evolution, and optimization.
• Experience with indexing and partitioning strategies.
• Understanding of ETL/ELT workflows and streaming data pipelines versus batch systems.
• Proficiency with GitLab and CloudWatch.

Preferred Skills

• Experience with Great Expectations or other data quality validation frameworks.
• AWS architecture experience.
• Familiarity with Schema Registry and message formats such as Avro or ORC.

Key Skills

Education

Any Graduate