Description

Lead automation testing and architect enterprise-wide initiatives for ETL testing, data migration, and transformation phases.

Responsibilities

• Architect and implement automation testing frameworks for ETL, data migration, and transformation processes.
• Debug, troubleshoot, and resolve complex technical issues within large-scale big data application deployments.
• Design and maintain CI/CD pipelines using GitLab.
• Brief stakeholders and senior management on the benefits and constraints of proposed technology solutions.
• Optimize data processing by implementing indexing and partitioning strategies.

Required Skills

• 6+ years of IT experience with a focus on testing and automation.
• Proficiency in Java, Python, and Groovy.
• Hands-on experience with AWS services, specifically S3, Glue, and Lambda.
• Expertise in Spark, Scala, and Java programming.
• Deep understanding of Databricks, Structured Streaming, Delta Lake, and Delta Live Tables.
• Knowledge of Data Lake concepts including time travel, schema evolution, and optimization.
• Advanced SQL skills covering joins, aggregations, windowing functions, CTEs, and RDBMS schema design.
• Experience with ETL/ELT workflows and tools like AWS Data Migration Service.
• Proficiency with GitLab and AWS CloudWatch for CI/CD and monitoring.

Preferred Skills

• Experience with Great Expectations or similar data quality validation frameworks.
• AWS architecture experience.

Key Skills

Education

Any Graduate