Description

Design and optimize scalable batch and real-time streaming data pipelines on Azure Databricks for medical device telemetry.

This role is hybrid.

Responsibilities

• Build and maintain production-grade ETL/ELT workflows for large-scale structured and unstructured datasets.
• Develop real-time telemetry ingestion solutions using Azure Event Hubs and Stream Analytics.
• Optimize Databricks workloads through Spark tuning, partitioning, caching, and Delta Lake strategies.
• Design and implement Data Services, APIs, and Microservices for downstream analytics and ML platforms.
• Troubleshoot production issues, perform root cause analysis, and ensure data governance and privacy compliance.

Required Skills

• 10+ years of overall IT experience with strong hands-on Azure Databricks expertise.
• Expert-level PySpark, Spark SQL, Delta Lake, and Spark performance tuning.
• Strong experience with Azure Cloud services for data engineering.
• Experience building ETL/ELT pipelines, Data Services, APIs, and Microservices.
• Expertise in CI/CD, DevOps practices, and production support.
• Experience handling structured and unstructured data in cloud and on-prem environments.
• Bachelor's degree in a related field.

Preferred Skills

• Experience in Medical Device or Healthcare domains.
• Knowledge of IoT & Telemetry Data Processing, Medallion Architecture, and ML Feature Pipelines.

Key Skills

Education

Bachelor's degree