Description

You will lead the design and implementation of Azure-based data engineering solutions and Lakehouse architectures.

This role is remote.

Responsibilities

• Lead solution design discussions and prepare technical documentation including HLD, LLD, and architecture diagrams.
• Mentor junior engineers and enforce coding guidelines, design patterns, and peer review processes.
• Design and maintain Lakehouse solutions using a bronze-silver-gold architecture with ADLS Gen2.
• Build modular, reusable orchestration workflows using Azure Data Factory and Databricks Workflows.
• Manage data governance and access control using Unity Catalog, including object lineage and Azure AD integration.

Required Skills

• 5+ years of experience with Azure Databricks, PySpark, Databricks Workflows, and Unity Catalog.
• 4+ years of experience with Azure Data Factory (ADF), ADLS Gen2, and Azure SQL.
• 3+ years of experience in Python programming, including package builds and dependency management.
• Expertise in Spark SQL tuning, file format optimization (Parquet/Delta), and partitioning.
• Hands-on experience with Azure networking and security, including VNets, Private Endpoints, Key Vaults, and Managed Identities.
• Proficiency in T-SQL, stored procedures, and metadata management on Azure SQL.
• Experience writing modular, testable Python code and implementing unit tests with Pytest.
• Knowledge of DevOps tools for deployment automation, such as Azure DevOps, ARM, Bicep, or Terraform.
• Strong version control skills using Git and familiarity with code quality tools like SonarQube or pylint.

Preferred Skills

• 10+ years of total working experience in data engineering.
• Experience with CI/CD pipelines and automation.

Key Skills

Education

Any Graduate