Description

• We are seeking a Senior Data Platform Engineer to support the administration, automation, governance, security, and optimization of our enterprise Databricks Lakehouse platform on AWS.
• This is a hands-on technical role responsible for maintaining and enhancing the Databricks platform, implementing infrastructure automation, enforcing compute and security standards, supporting cost-governance initiatives, and enabling reliable Data Engineering and Data Science workloads.
• The ideal candidate will have strong hands-on experience with Databricks platform administration, AWS cloud services, Terraform-based infrastructure automation, and DevOps practices. The individual will evaluate emerging Databricks capabilities through proof-of-concept implementations and contribute to reusable platform patterns and standards.
• This role will collaborate closely with Data Engineering, Data Science, Cloud Platform, Security, and Governance teams to ensure the Databricks environment remains secure, reliable, scalable, and cost-efficient.
• Hands-on experience administering Databricks workspaces, including cluster policies, job policies, compute governance, networking, security, and platform integrations
• Experience implementing Unity Catalog governance and access management
• Experience with Delta Lake architecture, performance optimization, and troubleshooting
• Strong hands-on experience with AWS IAM, VPC networking, S3, AWS KMS, CloudWatch, PrivateLink, cross-account access, and cloud security
• Experience using Terraform for Databricks and AWS infrastructure
• Experience with infrastructure-as-code design patterns and platform automation frameworks
• Experience with Databricks CLI, Databricks REST APIs, Python, and Bash scripting
• Experience developing CI/CD pipelines and automated deployment and release processes
• Experience with Git-based version control and development workflows
• Experience with monitoring, logging, observability, incident management, and root cause analysis
• Experience with Apache Spark, PySpark, SQL optimization, streaming frameworks, data pipeline orchestration, data modeling, and ETL best practices
• Databricks Certified Data Engineer or Databricks Platform Architect certification
• AWS Certified Solutions Architect certification
• Experience designing enterprise-scale Lakehouse platforms
• Experience supporting large-scale machine learning workloads
• Experience implementing data governance frameworks

What You Know:

• Minimum 8-10 years in Databricks Data Engineer
• Experience designing enterprise-scale Lakehouse platforms
• Experience supporting large-scale ML workloads
• Experience implementing data governance frameworks
• Experience in AWS IAM, VPC networking and security, S3 data lake architecture, AWS KMS encryption, CloudWatch monitoring, PrivateLink connectivity, cross-account access architecture, and enterprise cloud security practices, along with Terraform for Databricks and AWS infrastructure, Infrastructure as Code (IaC) design patterns,
• Experience in Databricks CLI and API automation, Python/Bash scripting, platform automation frameworks, CI/CD pipeline development, Git-based version control workflows, automated deployment and release processes, monitoring, logging, observability frameworks, incident management, root cause analysis, Apache Spark/PySpark, SQL optimization, Delta Lake performance tuning, streaming frameworks, data pipeline orchestration, and data modeling and ETL best practices

Key Skills

Education

Bachelor's degree