Description

You will lead the design and hands-on development of automated data lineage capture across complex hybrid environments.

Responsibilities

• Design and implement automated data lineage capture across cloud (Snowflake, AWS), legacy databases (SQL Server, Oracle), and NoSQL stores (MongoDB).
• Implement and customize open-source lineage frameworks (Spline, OpenLineage) for end-to-end lineage tracking.
• Build connectors and agents to bridge disparate systems to the central lineage framework.
• Apply AI/ML techniques, such as NLP for code parsing, to infer and enhance lineage automatically.
• Integrate the lineage pipeline with enterprise governance tools like Collibra to publish actionable lineage.

Required Skills

• 5+ years of experience delivering automated data lineage solutions in large-scale, hybrid environments.
• Deep technical expertise with Spline, OpenLineage, Marquez, or comparable open-source lineage frameworks.
• Strong proficiency in Python, Scala, or Java.
• Deep familiarity with SQL and experience analyzing query logs from systems like Snowflake, SQL Server, Oracle, and MongoDB.
• Demonstrated ability to apply AI/ML to metadata challenges, including code parsing or log pattern detection.
• Experience integrating lineage data with commercial governance catalogs like Collibra.
• Proven ability to move lineage implementation from concept to production-grade code at enterprise scale.

Key Skills

Education

Any Gradute