Description

You will act as a Databricks Test Engineer focused on validating data processing pipelines and ETL processes within an AWS environment.

Responsibilities

• Develop and execute test plans for ETL processes and data transformations on Databricks.
• Implement automated data quality tests using Python, PySpark, or frameworks like Great Expectations.
• Perform data profiling, anomaly detection, and reconciliation between source and target systems.
• Test data integration workflows involving AWS services including S3, Redshift, and RDS.
• Conduct performance testing of Databricks jobs and clusters to ensure scalability.
• Troubleshoot and resolve data quality issues in collaboration with data engineering teams.

Required Skills

• 8+ years of experience in the testing domain.
• 3+ years of experience in data testing or quality assurance within a data engineering environment.
• 2+ years of hands-on experience with Databricks and Apache Spark.
• Proficiency in SQL and Python/PySpark for data manipulation.
• Hands-on experience with AWS cloud services, specifically S3.
• Domain knowledge in Life and Annuity insurance.
• Strong understanding of ETL processes, data warehousing, and big data technologies.
• Experience with Databricks Lakehouse architecture and Delta Lake.
• Familiarity with CI/CD pipelines and DevOps practices relevant to data engineering.

Preferred Skills

• Certifications in Databricks, Apache Spark, or AWS.
• Experience with data orchestration tools like Airflow.

Key Skills

Education

Not specified