Description

- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.
- Build and manage Medallion Architecture (Bronze, Silver, Gold) for enterprise data platforms.
- Ingest data from various structured and unstructured sources including APIs, databases, files, ERP, CRM, and cloud storage.
- Optimize Spark jobs, cluster configurations, and data processing workloads for performance and cost efficiency.
- Develop reusable frameworks, libraries, and components for ingestion, transformation, and data quality validation.
- Implement CI/CD pipelines using Azure DevOps or Git-based deployment methodologies.
- Monitor, troubleshoot, and resolve production data pipeline issues.
- Collaborate with business analysts, architects, and stakeholders to translate business requirements into technical solutions.

### Microsoft Fabric (Secondary)

- Develop and maintain data solutions using Microsoft Fabric components including Data Factory, Lakehouse, Warehouse, Notebooks, and Data Pipelines.
- Support data integration and modernization initiatives from legacy platforms to Fabric.
- Enable Power BI semantic models and reporting solutions leveraging Fabric architecture.
- Implement governance, security, and data management best practices within Fabric.

### Databricks Genie (Secondary)

- Support implementation and adoption of Databricks Genie for conversational analytics and natural language querying.
- Collaborate with business users to define semantic layers and business-friendly data models.
- Enable self-service analytics through AI-powered data exploration capabilities.
- Assist in developing governance and controls for AI-driven data access.

## Required Technical Skills

- Azure Databricks
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Azure Data Lake Storage (ADLS)
- Azure Data Factory (ADF)
- Git / Azure DevOps
- Data Modeling
- ETL / ELT Development
- Performance Tuning & Optimization
- Data Warehousing Concepts

## Preferred Skills

- Microsoft Fabric
- Fabric Lakehouse & Warehouse
- Power BI
- Databricks Genie
- Unity Catalog
- Databricks Workflows
- REST APIs
- Data Governance & Security
- AI/GenAI-enabled Analytics Solutions

## Qualifications

- Bachelor's degree in Computer Science, Engineering, Information Technology, or related field.
- 7+ years of experience in Data Engineering.
- 4+ years of hands-on experience with Azure Databricks.
- Experience delivering enterprise cloud data platforms.
- Databricks certification (Associate/Professional) is a strong advantage

Key Skills

Education

Bachelor's degree