Description

You will manage and scale production data environments across public cloud infrastructures.

Responsibilities

• Setup and maintain production-scale Databricks environments on Azure and AWS.
• Manage large-scale data storage including ADLS Gen2 and AWS S3 for multi-tenant teams.
• Maintain microservices for job scheduling, security, financial, and administrative operations.
• Develop automation for configuration management, service deployments, monitoring, and alerting.
• Optimize cloud spend by monitoring resource usage and implementing cost-reduction tools.
• Ensure security and privacy compliance by implementing data lockdown solutions.

Required Skills

• 5+ years of experience in data platform or cloud engineering roles.
• Proficiency with AWS and Azure cloud infrastructure.
• Hands-on experience with ADLS Gen2 and AWS S3.
• Experience managing Databricks Workspace components including Unity Catalog, Vector Search, SQL Warehouse, and Spark workloads.
• Knowledge of Airflow, DAGs, and Kubernetes (AKS or EKS).
• Experience with Hive, MySQL, MongoDB, or Vector Databases.
• Ability to implement monitoring using Databricks System Tables, Prometheus, Splunk, ELK, or PowerBI.
• Experience troubleshooting Linux servers and Kubernetes environments.
• Proficiency with Jira and ServiceNow.

Preferred Skills

• Experience with Neo4J, Collibra, or Metric Insights.
• Familiarity with integrating GenAI technologies into data platforms.

Key Skills

Education

Any Graduate