Description

You will design and build Databricks-based lakehouse architectures on AWS, owning the end-to-end lifecycle from data engineering to ML model deployment.

Responsibilities

• Architect scalable lakehouse layers (Bronze/Silver/Gold) with clear separation of compute and serving layers.
• Develop, train, and optimize ML models using Python, PySpark, and MLflow, deploying them via Databricks Workflows.
• Build high-performance ETL/ELT pipelines using PySpark and SQL, implementing Delta Lake best practices like OPTIMIZE and ZORDER.
• Design low-latency API strategies and caching solutions where Spark is insufficient to accelerate reads and reduce costs.
• Translate ambiguous business requirements into robust, scalable analytical and ML architectures.

Required Skills

• 10+ years of experience in data engineering, ML engineering, or AI/ML architecture.
• Deep expertise in Databricks on AWS, including Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong proficiency in Python (pandas, numpy, scikit-learn) and PySpark/Spark SQL.
• Experience handling multi-terabyte structured time-series workloads in distributed environments.
• Proficiency with AWS services: S3, IAM, Glue Catalog, and secure networking.
• Strong understanding of ML algorithms, distributed systems, and data optimization strategies.
• Bachelor's degree in Computer Science, Data Science, Engineering, Statistics, or a related field.

Preferred Skills

• Familiarity with deep learning frameworks such as TensorFlow or PyTorch.
• Experience with CI/CD pipelines using GitHub Actions or GitLab CI.
• Proven track record in MLOps and production deployment of ML models.

Key Skills

Education

Bachelor's degree