Description

Key Responsibilities

• Design, develop, and maintain batch and streaming data pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Build data solutions using medallion architecture (Bronze, Silver, Gold).
• Develop and optimize dimensional data models and SQL workloads in Azure Synapse Analytics.
• Develop ETL processes to extract, transform, and load data from multiple sources.
• Orchestrate and monitor data workflows using Databricks Jobs/Workflows, Azure Data Factory, and Synapse pipelines.
• Implement scheduling, monitoring, alerting, and failure recovery for data pipelines.
• Apply data governance, security, access controls, data masking, and data quality standards.
• Support secure handling of sensitive data including PHI/PII in accordance with applicable policies and regulations.
• Participate in requirements gathering, solution reviews, and technical design activities.
• Collaborate with data architects, data scientists, analysts, solution leads, and project managers.
• Troubleshoot data integration, pipeline, quality, and performance issues.
• Develop technical documentation, data flow diagrams, architecture documentation, and implementation guidelines.
• Provide technical guidance and support to project teams and stakeholders.

Required Skills

• Strong experience in data engineering, data integration, and data pipeline development.
• Hands-on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience with Azure Synapse Analytics and SQL optimization.
• Experience developing batch and streaming data pipelines.
• Strong knowledge of ETL/ELT processes and data modeling.
• Experience with Azure Data Factory and/or Databricks Workflows.
• Strong SQL and relational database skills.
• Experience with data governance, security, data quality, and access controls.
• Experience working with sensitive data such as PHI/PII and understanding of HIPAA requirements.
• Strong analytical, troubleshooting, documentation, and communication skills.
• Ability to work effectively with both technical and non-technical stakeholders.

Preferred Skills

• Experience with Unity Catalog and role-based access controls.
• Experience with row-level and column-level security and data masking.
• Experience supporting healthcare or government data environments.
• Experience with data architecture and enterprise data strategy.
• Experience providing technical guidance or mentoring to other team members

Key Skills

Education

Bachelor's degree