Description

Key Skills: Pyspark, Azure ADF, Data Bricks, SQL, Azure Devops

Roles and Responsibilities:

• Design, develop, and deploy scalable data pipelines using Databricks and Azure Data Factory.
• Implement ETL/ELT processes to ingest, transform, and load data into data lakes and data warehouses.
• Write and optimize complex SQL queries and PySpark jobs for performance, scalability, and efficient data retrieval.
• Integrate structured, semi-structured, and unstructured data sources, including batch and streaming ingestion patterns.
• Apply data governance and quality practices, and monitor/troubleshoot pipelines to ensure data integrity and availability.

Skills Required:

• 4 - 10 years of relevant data engineering experience
• Hands-on experience with Databricks including PySpark, Serverless SQL, Unity Catalog, Lakehouse, and Medallion Architecture
• Proficiency with Azure Data Factory, Azure Data Lake Storage, and Azure SQL Database
• Strong expertise in writing and optimizing complex SQL queries
• Experience implementing ETL/ELT workflows and data modeling best practices, with governance and security using Azure Key Vault

Good to Have:

• Azure DevOps for CI/CD and operational support of data pipelines

Education: Any Graduation, Any Post Graduation

Key Skills

Education

Any Graduate