Description

Design and maintain scalable, secure Databricks environments on Azure.

Responsibilities

• Design and implement Databricks clusters and tailored data solutions, including API ingestion and pipelines.
• Build data pipelines using medallion architecture to clean, transform, and aggregate data from disparate sources.
• Manage role-based access and data security using Unity Catalog.
• Oversee on-prem data ingestion into Azure using Azure Data Factory (ADF).
• Provide technical mentorship, escalated troubleshooting support, and cluster health reviews.

Required Skills

• 5+ years implementing Lakehouse architecture using Databricks Data Engineering, SQL Analytics, Delta Lake, and Unity Catalog.
• 6+ years proficiency in Python, SQL, and PySpark for data integration and performance tuning.
• 7+ years experience with SQL Server features, including SSIS and CDC.
• 4+ years configuring Azure infrastructure and building large ingestion pipelines via ADF.
• 5+ years expertise in Relational and Dimensional modeling.
• 2+ years experience with Databricks platform security and data access control mechanisms.
• 2+ years implementing serverless real-time or near real-time architectures using Cloud (Azure, AWS, or GCP) and Spark Streaming/ML.
• Experience with GIT code versioning.
• Bachelor's degree in Computer Science.

Preferred Skills

• Local to the Triangle, NC region and able to come onsite for equipment collection and monthly meetings.

Key Skills

Education

Bachelor's degree in Computer Science