Description

Key Responsibilities
 
Application Development: Design, code, and deploy custom, interactive web applications and data UIs using Databricks Apps to expose analytics and ML tools directly to business users.
Pipeline Engineering: Write modular, highly optimized PySpark and structured SQL code within Delta Live Tables (DLT) and Databricks Workflows.
UI/UX Integration: Integrate data pipelines with popular Python-based application frameworks like Streamlit, Dash, or Gradio hosted securely inside Databricks.
Performance Tuning: Optimize code execution paths, query structures, and file layouts using modern platform features like Liquid Clustering (CLUSTER BY).
DevOps & App Lifecycle: Package, deploy, and manage the CI/CD lifecycle of both pipelines and Databricks Apps using Databricks Asset Bundles (DABs) and Git.
Data Security: Ensure all developed applications and data models conform to governance, access controls, and security policies managed by Unity Catalog.
ML Integration: Build application interfaces that interact with production machine learning models served via MLflow endpoints.
 
Required Skills and Qualifications
Core Expertise: Deep development experience within the Databricks platform environment and Apache Spark framework.
Languages: Advanced programming skills in Python (PySpark, Pandas) and highly structured SQL.
App Frameworks: Proven experience building web applications using Streamlit, Dash, or Gradio, with an understanding of state management and API integration.
Storage Mechanics: Solid understanding of Delta Lake architecture, table optimization, and modern dynamic data layouts.
SDLC Tools: Proficiency with Git, CI/CD pipelines, and infrastructure-as-code deployment via Databricks Asset Bundles (DABs).
Certifications (Preferred): Databricks Certified Data Engineer Associate/Professional or Databricks Certified Developer

Key Skills

Education

Bachelor's degree