Description

You will lead the design and implementation of Azure-based data engineering solutions and Lakehouse architectures.

Responsibilities

• Lead solution design discussions and prepare technical documentation including HLD, LLD, and architecture diagrams.
• Mentor junior engineers and enforce coding guidelines, design patterns, and peer review processes.
• Design and maintain Lakehouse solutions using a bronze-silver-gold architecture with ADLS Gen2.
• Build modular, reusable orchestration workflows using Azure Data Factory and Databricks Workflows.
• Manage data governance and access control using Unity Catalog, including object lineage and Azure AD integration.
• Optimize Spark jobs and large-scale data processing using RDD/DataFrame APIs and partitioning strategies.

Required Skills

• 5+ years of experience with Azure Databricks, PySpark, Databricks Workflows, and Unity Catalog.
• 4+ years of experience with Azure Data Factory (ADF), ADLS Gen2, and Azure SQL.
• 3+ years of experience in Python programming, including package builds and dependency management.
• Expertise in Spark SQL tuning, file format optimization (Parquet/Delta), and partitioning.
• Hands-on experience with Azure networking and security, including VNets, Private Endpoints, Key Vaults, and Managed Identities.
• Proficiency in T-SQL, stored procedures, and metadata management on Azure SQL.
• Experience writing modular, testable Python code and implementing unit tests with Pytest.
• Knowledge of DevOps tools for deployment automation, such as Azure DevOps, ARM, Bicep, or Terraform.
• Strong version control skills using Git and familiarity with code quality tools like SonarQube or pylint.

Key Skills

Education

Any Graduate