Description

Key Skills: Azure Databricks, PySpark, Delta Lake, T-SQL, Azure Data Factory, Unity Catalog, Azure SQL, Python, ETL/ELT, Azure Cloud

Good to Have Skills: Healthcare domain knowledge, Scala, ADLS Gen2, Microsoft TFS, Visual Studio, DevOps, SSIS, Data Lake, Data Storage, SOLID development principles, problem-solving skills, team collaboration, time management, and communication skills.

Roles & Responsibilities:

• Work with different scrum teams to develop all quality database programming requirements of the sprint.
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.
• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.
• Implement Unity Catalog for centralized data governance, access control, and data lineage tracking.
• Independently analyze, solve, and correct issues in real time, providing problem resolution end-to-end.
• Develop unit tests to be able to test them automatically for quality assurance.
• Use SOLID development principles to maintain data integrity and cohesiveness across systems.
• Interact with product owner and business representatives to determine and satisfy business needs.

Experience Required: 7-10 years of experience as a Lead Data Engineer with proven experience in T-SQL programming, Azure Databricks, Spark (PySpark/Scala), Delta Lake, Unity Catalog, and ADLS Gen2. Experience with cloud platforms such as Azure and exposure to Microsoft TFS, Visual Studio, and DevOps is required

Key Skills

Education

Any Graduate