Description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
• Develop and manage Delta Lake-based Lakehouse solutions following Medallion Architecture (Bronze, Silver, Gold).
• Integrate data from multiple sources including databases, APIs, cloud storage, and enterprise applications.
• Create and maintain data ingestion frameworks using Azure Data Factory (ADF).
• Optimize Spark jobs, cluster configurations, and storage structures to improve performance and reduce costs.
• Implement data quality, validation, monitoring, logging, and alerting mechanisms.
• Collaborate with Data Architects and Business Analysts to gather and translate business requirements into technical solutions.
• Implement security, governance, and access controls using Unity Catalog and Azure security services.
• Support CI/CD deployment processes using Azure DevOps, Git, and automated release pipelines.
• Participate in code reviews, design discussions, and knowledge-sharing initiatives.
• Troubleshoot production issues and provide operational support for critical data pipelines.

Mandatory Skills

Databricks & Big Data

• Azure Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Databricks Workflows
• Unity Catalog

Azure Data Services

• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• Azure Key Vault
• Azure Synapse Analytics

Programming & Database

• Python
• SQL / T-SQL
• Data Modeling
• Data Warehousing Concepts

DevOps & Version Control

• Azure DevOps
• Git/GitHub
• CI/CD Pipelines

Preferred Skills

• Snowflake
• Microsoft Fabric
• Kafka/Event Streaming
• Power BI
• Terraform/IaC
• Azure Functions
• REST API Integration
• Data Governance and Metadata Management

Desired Experience

• 6+ years of overall Data Engineering experience.
• Minimum 3+ years of hands-on experience with Azure Databricks and PySpark.
• Experience building enterprise-grade Lakehouse architectures.
• Strong understanding of performance tuning and optimization techniques for Spark workloads

Key Skills

Education

Bachelor's degree