Description

You will design and optimize scalable ETL pipelines to process large-scale structured and semi-structured datasets within Azure environments.

Responsibilities

• Build and maintain automated, modular ETL workflows using Azure Data Factory and Databricks notebooks for batch and near real-time integration.
• Implement scalable data transformations in Databricks using PySpark and SQL, optimizing through partitioning and cluster configurations.
• Design backend data lake architectures to ensure data lineage, traceability, and reusability for downstream applications.
• Integrate data from external systems, including SAP, using middleware solutions like MuleSoft.
• Secure healthcare datasets by implementing encryption, tokenization, and role-based access control for PHI/PII compliance.
• Establish monitoring and error-handling mechanisms using Azure Monitor, Log Analytics, and Python scripts.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and PySpark.
• Expertise in Azure Databricks and Databricks notebooks.
• Hands-on experience with Azure Data Factory.
• Strong SQL skills for data transformation.
• Experience with Delta Lake and Azure Data Lake Storage.
• Knowledge of Azure Event Hubs and Azure Key Vault.
• Familiarity with MuleSoft for SAP system integration.
• Bachelor’s degree in Computer Science, IT, or a related field.

Key Skills

Education

Any Graduate