Description

You will design, build, and maintain end-to-end MLOps pipelines on Databricks running on AWS. You own the lifecycle of machine learning models from experiment tracking to production deployment and monitoring.

Responsibilities

• Design and automate ML workflows covering data preparation, feature engineering, training, and evaluation using Databricks Workflows and Delta Lake.
• Deploy and manage ML models using MLflow Model Registry and Databricks Model Serving, ensuring reliable production releases.
• Build and maintain CI/CD pipelines for Databricks and ML workloads using Git, Bitbucket, Jenkins, and Databricks Asset Bundles.
• Implement Infrastructure as Code with Terraform to provision and manage Databricks and AWS resources securely.
• Monitor model health, detect data drift, and implement automated retraining strategies to maintain prediction quality.

Required Skills

• 10+ years of software engineering experience with a strong focus on MLOps and distributed systems.
• Proficiency in Python, PySpark, and SQL for developing and operationalizing machine learning solutions.
• Hands-on experience with Databricks Machine Learning, including MLflow for experiment tracking and model lifecycle management.
• Experience with AWS services: S3, IAM, EC2, Lambda, ECR, ECS/EKS, CloudWatch, and Secrets Manager.
• Strong understanding of Apache Spark architecture, optimization, and distributed data processing.
• Experience implementing CI/CD pipelines using Git, Bitbucket, Jenkins, and Databricks Asset Bundles (DAB).
• Proficiency with Terraform for infrastructure automation.
• Knowledge of Unity Catalog for governance, security, and access management.

Preferred Skills

• Experience with batch inference pipelines and model serving optimization.
• Strong understanding of MLOps best practices including reproducibility, versioning, and testing.

Key Skills

Education

Bachelor's degree