Description

Key Skills and Experience

·      Databricks Ecosystem: Hands-on experience with Databricks Workspace, Unity Catalog, Delta Lake, Spark UI, and MLflow.

·      Programming & Data Engineering: Strong development experience with Python, PySpark, SQL, and/or Java, including the development and optimization of data processing and ETL/ELT pipelines.

·      Cloud Platforms: Strong Microsoft Azure experience, particularly Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Data Factory. Experience with AWS, including S3 and IAM, is also desirable.

·      Apache Spark: Experience developing, troubleshooting, and performance-tuning Spark-based workloads and data processing pipelines.

·      DevOps & MLOps: Experience with Git, CI/CD pipelines, OpenShift, Kubernetes, and Terraform / Infrastructure as Code (IaC).

·      Security & Governance: Experience implementing or supporting Databricks security, access controls, secrets management, identity integration, data governance, and Unity Catalog.

·      Automation & Operations: Experience with Databricks workflows, job orchestration, monitoring, troubleshooting, performance optimization, and production support.

·      Enterprise Delivery: Ability to collaborate across technical teams and operate within established enterprise architecture, security, change-management, and governance standards.

·      Knowledge of Apache Airflow

Preferred Experience

·      Experience designing and delivering production-grade Databricks solutions in a large enterprise environment, particularly where security, data governance, resiliency, automation, and operational support are key requirements

Key Skills

Education

Bachelor's degree