Description

Required Skills

• Strong hands-on experience with Databricks Machine Learning
• Proficiency in Python, PySpark, and SQL for developing and operationalizing machine learning solutions.
• Hands-on experience with MLflow for experiment tracking, model registry, model versioning, and model lifecycle management.
• Experience deploying and managing machine learning models using Databricks Model Serving and batch inference pipelines.
• Strong understanding of the end-to-end ML lifecycle, including feature engineering, model training, validation, deployment, monitoring, and retraining.
• Experience building scalable ML pipelines using Databricks Workflows and Delta Lake.
• Hands-on experience with AWS services such as S3, IAM, EC2, Lambda, ECR, ECS/EKS, CloudWatch, and Secrets Manager.
• Experience implementing CI/CD pipelines for Databricks and ML workloads using Git, Bitbucket, Jenkins, and Databricks Asset Bundles (DAB).
• Experience with infrastructure automation using Terraform (Infrastructure as Code).
• Strong understanding of Apache Spark architecture, optimization, and distributed data processing.
• Experience working with Unity Catalog for governance, security, and access management.
• Knowledge of model monitoring, data drift detection, model performance monitoring, and automated retraining strategies.
• Understanding of MLOps best practices, including reproducibility, versioning, testing, and governance.
• Strong collaboration skills to work with Data Scientists, Data Engineers, and Platform Engineering teams.
• Excellent analytical, problem-solving, and communication skills.

Roles & Responsibilities

• Design, develop, and maintain end-to-end MLOps pipelines on Databricks running on AWS.
• Build and automate machine learning workflows covering data preparation, feature engineering, model training, evaluation, deployment, and monitoring.
• Deploy and manage ML models using MLflow Model Registry and Databricks Model Serving.
• Develop and maintain CI/CD pipelines for ML solutions across development, staging, and production environments.
• Collaborate with Data Scientists to productionize machine learning models and ensure reliable deployments.
• Monitor model health, prediction quality, data drift, and system performance, and implement retraining strategies where required.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Implement Infrastructure as Code (Terraform) for provisioning and managing Databricks and AWS resources.
• Ensure platform security, governance, and compliance using Unity Catalog and AWS IAM.
• Troubleshoot production issues, perform root cause analysis, and continuously improve platform reliability.
• Document MLOps processes, deployment standards, and operational best practices

Key Skills

Education

Bachelor's degree