Description

You will design, develop, and maintain data pipelines using Azure Databricks. You will implement ETL/ELT workflows to process large-scale structured and unstructured data.

Responsibilities

• Optimize Spark jobs for performance, scalability, and cost.
• Develop and manage Databricks notebooks using PySpark, Scala, and SQL.
• Integrate Databricks with Azure services including Azure Data Factory, Azure Synapse Analytics, and Azure Key Vault.
• Work with Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, and Azure SQL.
• Implement data quality, validation, and monitoring frameworks.

Required Skills

• 5+ years of experience in data engineering or pipeline development.
• Proficiency in PySpark, Scala, and SQL.
• Hands-on experience with Azure Databricks and ETL/ELT workflows.
• Experience with Azure Data Factory (ADF).
• Knowledge of Azure Data Lake Storage (ADLS Gen2) and Azure Blob Storage.
• Understanding of Azure SQL and Azure Synapse Analytics.

Key Skills

Education

Any Gradute