Description

Lead the design and implementation of scalable data and machine learning platforms using Databricks on AWS.

Responsibilities

• Architect Databricks-based lakehouse systems on AWS, implementing Delta Lake, S3, and Unity Catalog.
• Develop, train, and optimize ML models using Python, PySpark, MLflow, and Databricks Machine Learning.
• Design and build scalable ETL/ELT pipelines with PySpark, SQL, and Databricks Workflows.
• Deploy ML models into production via MLOps pipelines, ensuring separation of compute and serving layers.
• Optimize large-scale pipelines through caching, partitioning, ZORDER, and Spark tuning for multi-terabyte datasets.

Required Skills

• 10+ years in data engineering, ML engineering, or AI/ML architecture.
• Deep expertise in Databricks on AWS, including PySpark, Spark SQL, and Databricks Notebooks.
• Strong proficiency in Python, including pandas, numpy, and scikit-learn.
• Hands-on experience with Delta Lake best practices, Unity Catalog, and MLflow.
• Proficient in SQL and designing Bronze/Silver/Gold lakehouse layers.
• Experience with AWS services: S3, IAM, Glue Catalog, and networking.
• Ability to handle multi-terabyte structured time-series workloads.
• Bachelor’s or Master’s in Computer Science, Data Science, Engineering, Statistics, or related field.

Preferred Skills

• Experience with MLOps and production deployment pipelines.
• Understanding of CI/CD using GitHub Actions, GitLab CI, or similar.

Key Skills

Education

Any Graduate