Description

Build and maintain an end-to-end data quality framework to improve data capture, accuracy, and lineage.

Responsibilities

• Implement DQ rule engine solutions using Spark and Scala for real-time streaming and micro-batch processing.
• Develop tools to trace data lineage and provide visibility into source-to-target mapping throughout the pipeline.
• Build and maintain monitoring dashboards to track latency and ensure efficient data transformation.
• Identify and resolve data capture issues before collection and delivery.
• Collaborate with cross-functional teams to establish effective monitoring and notification mechanisms.

Required Skills

• 5+ years of experience in senior developer roles focusing on data quality frameworks and pipeline architectures.
• Proficiency in Scala, Python, Java, and SQL.
• Hands-on experience with Spark and Scala for real-time streaming and micro-batch processing.
• Experience with Azure or AWS cloud platforms for data processing and storage.
• Knowledge of data governance, data cataloging, and data lineage concepts.
• Bachelor's or Master's degree in Computer Science or a related field.

Key Skills

Education

Bachelor's or Master's degree in Computer Science