Description

Manage the Databricks platform on AWS, focusing on administration, deployment, and optimization for data engineering and machine learning workloads.

Responsibilities

• Configure, deploy, and maintain Databricks clusters and workspaces using Terraform to treat infrastructure as code.
• Monitor cluster performance, resource utilization, and auto-scaling to ensure cost-effectiveness and optimal performance.
• Implement and enforce security policies, access controls, and encryption mechanisms to protect sensitive data.
• Manage platform feature releases, software updates, and backup/disaster recovery strategies.
• Collaborate with data engineers and scientists to integrate Databricks with existing data lakes and warehouses.

Required Skills

• 7+ years of related professional experience.
• 3 years of experience with Databricks Lakehouse Platform Architecture on AWS.
• Proficiency with Terraform HCL for infrastructure provisioning.
• Strong command of SQL and Python.
• Hands-on experience with Apache Spark.
• Experience managing GitLab CI/CD pipelines.
• Knowledge of MLOps practices.
• Working knowledge of AWS cloud platforms.
• Bachelor's degree in a relevant field such as computer science or data engineering.

Preferred Skills

• Experience within the Utility and Energy sectors.

Key Skills

Education

ANY GRADUATE