Description

Key Responsibilities 

Passport File Ingestion and Validation 

• Ingest multiple passport files received for asset deployment or update. 
• Validate each passport file to ensure all required deployment data is present. 
• Scan the passport file format and register valid passport files for processing. 
• Generate validation reports in JSON format. 

Dependency DAG and Execution Order 

• Analyze dependencies between assets across multiple passport files. 
• Build an asset dependency DAG based on upstream and downstream relationships. 
• Implement topological sorting to determine the safe asset deployment execution order. 
• Ensure downstream assets wait until all required upstream dependencies are successfully completed. 
• Handle success dependency, control flow, data/artifact flow, and loop execution per passport file. 

Asset Deployment Loop 

• Build the full deployment loop for each passport file in DAG execution order. 
• Extract changes from the passport file by comparing with previously deployed assets. 
• Support change detection for new, changed, and existing assets. 
• Generate required DDL scripts and deployment logs. 
• Deploy or update DDL changes into Databricks and Unity Catalog. 
• Ensure each passport file completes the full deployment loop before dependent assets are triggered. 

Downstream System Integration 

Develop integrations with the following enterprise systems: 

• Databricks 
• Unity Catalog 
• Alation 
• SharePoint 
• Knowledge Graph platforms 
• Vector databases 
• Ontology or semantic model platforms 
• REST APIs and metadata services 

Build payload generation logic for each downstream system, including: 

• Alation metadata payloads 
• SharePoint document payloads 
• Knowledge graph payloads 
• Vector index payloads 
• Ontology or semantic model payloads 
• Implement parallel downstream deployment where applicable, while ensuring dependency and success/failure rules are respected. 

Post-DDL Deployment Processing 

• Extract changes to push to Alation only after successful DDL deployment. 
• Extract changes to push to SharePoint only after successful DDL deployment. 
• Extract changes to push to downstream systems only after successful DDL deployment. 
• Support downstream updates for: 
  • Alation metadata update or upsert 
  • SharePoint document and metadata update 
  • Knowledge Graph ingest or graph update 
  • Vector Database vector upsert and indexing 
  • Optional Ontological Model update and relationship management 

Failure Handling and Auditability 

• Stop the asset deployment loop if any required extraction or deployment step fails. 
• Prevent downstream deployment when upstream deployment is unsuccessful. 
• Generate deployment logs, payload files, validation evidence, and audit artifacts. 
• Maintain traceability of generated artifacts for every passport file and deployment step. 

Required Skills 

• Strong software engineering experience using Python or a similar backend programming language. 
• Experience building configuration-driven automation using JSON and YAML. 
• Strong understanding of DAGs, dependency resolution, graph processing, and topological sorting. 
• Experience developing CI/CD or deployment automation workflows. 
• Hands-on experience with Databricks and Unity Catalog deployment automation. 
• Experience integrating with enterprise platforms using REST APIs and metadata services. 
• Ability to build payload generation logic for multiple downstream systems. 
• Strong knowledge of validation, logging, error handling, audit tracking, and deployment status management. 
• Experience working with Git-based source control and environment-based deployments

Key Skills

Education

Bachelor's degree