Description

Responsibilities:

• Design, develop, and implement data processing pipelines using Azure Databricks and Spark.

• Write efficient and optimized code in Python (PySpark) and SQL for large-scale data processing.

• Architect and maintain data Lakehouse solutions leveraging Microsoft Fabric for analytics and reporting.

• Implement data governance and cataloging using Microsoft Purview, ensuring compliance and lineage tracking.

• Build and optimize ETL workflows for ingestion, transformation, and integration across Azure services.

• Ensure data quality, security, and integrity through robust validation and governance practices.

• Collaborate with cross-functional teams to understand business requirements and deliver scalable solutions.

• Maintain version control using Git and follow best practices for CI/CD in data engineering.

• Create comprehensive technical documentation, including requirements, design, and testing artifacts.

Requirements:

• 5+ years of hands-on experience in development and production implementation with Azure Databricks and Spark.

• Proven experience with Microsoft Purview for data governance and compliance.

• Working knowledge of Microsoft Fabric for analytics and integration.

• Proficiency in Python (PySpark) and SQL coding.

• Strong understanding of data lake architecture, ETL processes, and data quality management.

• Experience with Azure data analytics services (e.g., Data Lake, Synapse, Data Factory).

• Excellent technical writing and communication skills.

• Ability to optimize performance for large-scale data processing and ensure scalability

Key Skills

Education

Bachelor's degree