← Back to jobs

ConglomerateIT Logo
Data Quality Engineer

ConglomerateIT

 

Dallas, TX, USA

Posted On: Just posted
Experience: 7+ years
Availability: Onsite
Openings: 1
Category: Data Quality Engineer
Tenure: Contract - Corp-to-Corp
Related Jobs

No related jobs found

Description

Key Responsibilities

Design and execute comprehensive test strategies for batch processing, streaming pipelines, ETL/ELT workflows, and data analytics solutions.

Develop automated data quality checks to identify missing records, duplicate entries, schema changes, inconsistencies, and referential integrity issues.

Validate business rules, data transformations, aggregations, and processing logic using SQL and Python.

Test Databricks-based data engineering workflows, including Apache Spark applications developed with PySpark or Scala.

Verify data accuracy and traceability across source systems, intermediate processing layers, and downstream data consumers.

Validate AWS-based data pipelines built using services such as S3, Glue, Lambda, EMR, Step Functions, Redshift, Athena, Kinesis, and DynamoDB.

Test Kafka streaming applications to ensure data integrity, event ordering, and appropriate message delivery guarantees.

Validate Kafka producers, consumers, serialization formats, topic configurations, partitioning, offsets, retention settings, and schema evolution.

Assess pipeline behavior under duplicate events, delayed messages, consumer failures, reprocessing scenarios, backfills, and historical data loads.

Develop and maintain reusable Python-based test automation frameworks, validation utilities, test fixtures, and synthetic datasets.

Integrate automated data quality tests into CI/CD workflows for pre-merge checks, scheduled executions, and post-deployment verification.

Implement automated notifications and reporting mechanisms to identify data quality failures and pipeline anomalies.

Evaluate data pipeline scalability and performance under high-volume workloads, including throughput, latency, concurrency, and resource utilization.

Test failure recovery, retry mechanisms, idempotency, error handling, regression scenarios, and system resilience.

Monitor pipeline health using CloudWatch, Prometheus, Grafana, or comparable monitoring and observability platforms.

Validate data security controls, IAM permissions, encryption configurations, storage lifecycle policies, and access restrictions.

Collaborate with engineering teams to establish data quality SLAs and SLOs and support incident investigations, root-cause analysis, and post-incident reviews.

Required Qualifications

7+ years of experience in Quality Assurance, Software Development Engineer in Test (SDET), Data Quality Engineering, or related testing roles.

Strong hands-on expertise in SQL for complex data analysis, reconciliation, and validation.

Proficiency in Python for test automation, data processing, and quality assurance.

Proven experience testing data pipelines, ETL/ELT processes, and distributed data processing workflows.

Practical experience with Apache Kafka or other event-streaming technologies.

Strong understanding of AWS data services, including S3, Glue, Redshift, Lambda, Athena, or equivalent platforms.

Experience with Databricks and Apache Spark-based data processing is highly desirable.

Knowledge of batch and streaming architectures, data transformations, schema validation, and end-to-end data lineage.

Experience testing large-scale datasets and distributed systems.

Strong troubleshooting, analytical thinking, debugging, and problem-solving capabilities.

Ability to collaborate effectively with cross-functional engineering and analytics teams

Education

Bachelor's degree

Related Jobs

No related jobs found

← Back to jobs