Description

You will design and maintain data pipelines on the Databricks Lakehouse Platform, integrating agentic AI use cases into analytics workflows.

This role is remote.

Responsibilities

• Develop and optimize Databricks notebooks, workflows, and Delta Lake structures using Python and SQL.
• Implement Unity Catalog governance and manage semantic layers for analytics-ready datasets.
• Integrate LLM-enabled features and prompt engineering into data interaction tools like Databricks Genie.
• Manage CI/CD pipelines, version control with Git, and job scheduling, monitoring, and logging.
• Deploy and maintain Databricks infrastructure on AWS, Azure, or GCP.

Required Skills

• 10+ years of experience in data or analytics engineering.
• Deep expertise with the Databricks Lakehouse Platform, including Delta Lake and Unity Catalog.
• Strong proficiency in Python and SQL for data transformation and query optimization.
• Hands-on experience with CI/CD practices, Git version control, and job scheduling tools.
• Practical exposure to LLMs, prompt engineering, and AI-driven data interaction.
• Familiarity with Databricks Genie or similar conversational analytics interfaces.
• Experience deploying Databricks workloads on AWS, Azure, or GCP.

Preferred Skills

• Databricks certification.
• Understanding of data modeling basics.

Key Skills

Education

Bachelor's degree