Description

You will build and maintain data pipelines and AI solutions on Azure Databricks.

This role is on-site.

Responsibilities

• Develop and optimize PySpark, Python, and SQL data pipelines using Delta Lake.
• Integrate AI/ML solutions, including Generative AI, LLMs, and Azure OpenAI.
• Design ETL workflows using Azure Data Factory (ADF) or similar tools.
• Apply Spark optimization techniques to ensure efficient data processing.
• Collaborate on CI/CD practices and maintain robust cloud infrastructure.

Required Skills

• 5+ years of hands-on experience with Azure Databricks.
• Strong proficiency in PySpark, Python, and SQL.
• Experience with Delta Lake and Spark optimization.
• Knowledge of Azure Data Factory (ADF) or equivalent ETL tools.
• Familiarity with cloud platforms, specifically Azure.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with RAG, Vector Databases, or AI application development.
• Azure certifications.
• Exposure to MLflow or other AI frameworks.

Key Skills

Education

Bachelor's degree