Description

You will own the design and maintenance of end-to-end data pipelines, integrating data quality, cataloging, and GenAI capabilities.

Responsibilities

• Design and implement automated data quality components, including checks, alerts, and metric monitoring.
• Architect a data catalog to manage metadata and data structures across all pipelines.
• Develop APIs to facilitate data exchange between schedulers, observability systems, and GenAI Copilot functionality.
• Maintain full-stack code for pipelines, APIs, and quality components to support data scientists and analysts.

Required Skills

• 13+ years of professional experience in data engineering or related roles.
• Expertise in Python as the primary programming language.
• Experience with workflow orchestration tools such as Airflow or Luigi.
• Proficiency with data quality tools like Great Expectations or Apache Spark.
• Hands-on experience with data governance and cataloging platforms like Apache Atlas or Collibra.
• Strong background in RESTful API design and development.
• Familiarity with observability and monitoring systems like Prometheus or Grafana.
• Working knowledge of Java or Scala.
• Experience with cloud platforms including AWS, GCP, or Azure.

Preferred Skills

• Experience with GenAI or AI development platforms.
• Foundational knowledge of machine learning.
• Experience with data warehousing using Snowflake or Redshift.

Key Skills

Education

Any Graduate