Description

You will design, build, and maintain scalable data pipelines using PySpark and Azure Synapse Analytics. You are responsible for ensuring data reliability through comprehensive unit testing and rigorous debugging. Your role involves optimizing complex SQL queries and managing identity access via Azure Active Directory. You will collaborate with stakeholders to translate business requirements into technical solutions while addressing technical debt and improving system performance.

Responsibilities

• Design and code stable, scalable data pipelines integrating into existing workflows.
• Perform unit testing and debugging to ensure timely delivery of functional systems.
• Implement data quality frameworks and monitoring tools to maintain reliability.
• Refactor code to resolve defects, optimize performance, and reduce technical debt.
• Provide technical support and troubleshoot issues to ensure smooth operation.

Required Skills

• 10+ years of hands-on experience with Azure Data Engineering services.
• Strong proficiency in PySpark for large-scale data processing.
• Experience with Azure Synapse Analytics for data warehousing and analytics.
• Deep understanding of Azure Active Directory (AAD) and Active Directory concepts.
• Expertise in Advanced SQL for complex queries and performance optimization.
• Proven track record building and maintaining scalable data pipelines.
• Knowledge of data integration, transformation, and validation techniques.

Preferred Skills

• Experience with data quality frameworks and monitoring tools.
• Background in engineering or IT specialization.

Key Skills

Education

Any Graduate