Description

You own the automation pipeline that ingests, validates, and deploys asset configurations across Databricks and downstream enterprise systems.

This role is on-site.

Responsibilities

• Ingest and validate passport files, generating JSON validation reports and registering them for processing.
• Build asset dependency DAGs using topological sorting to determine safe deployment execution orders.
• Execute the full deployment loop, extracting changes, generating DDL scripts, and updating Databricks Unity Catalog.
• Develop integrations with Alation, SharePoint, Knowledge Graphs, Vector databases, and REST APIs to generate and push metadata payloads.
• Implement failure handling, audit logging, and traceability for all deployment steps and generated artifacts.

Required Skills

• 5+ years of software engineering experience with Python or similar backend languages.
• Strong understanding of DAGs, dependency resolution, graph processing, and topological sorting.
• Hands-on experience with Databricks, Unity Catalog, Delta Lake, and Spark.
• Experience building configuration-driven automation using JSON and YAML.
• Proven track record developing CI/CD or deployment automation workflows.
• Ability to build payload generation logic for multiple downstream systems via REST APIs.
• Strong knowledge of validation, logging, error handling, and Git-based source control.

Preferred Skills

• Experience with Alation metadata services and ontology or semantic model platforms.

Key Skills

Education

Bachelor's degree