Description

Key Responsibilities

Data Engineering Design and develop scalable data pipelines using Databricks, PySpark, Delta Lake, and Spark SQL

. Build ELT/ETL solutions for structured and unstructured data. Implement Medallion Architecture (Bronze, Silver, Gold layers)

. Optimize Databricks jobs, clusters, and workflows for performance and cost efficiency. Manage data ingestion from multiple cloud and on-premise sources

. AI & Machine Learning Develop and deploy Machine Learning models using Databricks ML and Python

. Implement MLOps practices for model training, deployment, monitoring, and governance

. Build predictive analytics solutions and recommendation engines

. Create feature stores and reusable ML pipelines. Generative AI Design and develop GenAI applications using: Large Language Models (LLMs) Retrieval Augmented Generation (RAG) Vector Databases AI Agents and Multi-Agent Frameworks Integrate Databricks Mosaic AI, LangChain, LangGraph, and OpenAI/Azure OpenAI services. Fine-tune and evaluate foundation models. Develop enterprise chatbots, knowledge assistants, and intelligent automation solutions. Cloud & Architecture Implement solutions on Azure, AWS, or GCP. Design secure and scalable Lakehouse architectures. Work with Unity Catalog, Delta Sharing, and data governance frameworks. Collaborate with business stakeholders, architects, and data scientists to deliver AI-driven business solutions. Required Skills Technical Skills Databricks Lakehouse Platform PySpark, Spark SQL Python Delta Lake Databricks Workflows Databricks Unity Catalog Azure Data Factory / AWS Glue SQL and Data Modeling Git, CI/CD, DevOps AI & GenAI Skills Machine Learning Deep Learning LLMs RAG Architecture Prompt Engineering LangChain / LangGraph Vector Databases (FAISS, Pinecone, ChromaDB) Databricks Mosaic AI MLflow Cloud Platforms Microsoft Azure (Preferred) AWS Google Cloud Platform Qualifications Bachelor's or Master's degree in Computer Science, Data Science, Engineering, Information Technology, or related field. 6+ years of experience in Data Engineering. 3+ years of experience with Databricks. Experience delivering AI/ML or Generative AI solutions in production environments. Databricks Certified Data Engineer or Machine Learning certification preferred. Preferred Experience Healthcare, Life Sciences, Banking, Retail, or Manufacturing domain experience. Experience with Azure OpenAI Service. Knowledge of Agentic AI and AI Governance. Experience in enterprise-scale data modernization programs. Key Deliverables Scalable Databricks data pipelines. Enterprise AI and GenAI solutions. Real-time analytics and AI-powered insights. AI-enabled automation and decision-support systems. Secure and governed Lakehouse architecture

Key Skills

Education

Bachelor's or Master's degrees