Description

Key Responsibilities

• Develop and maintain data pipelines using Azure Databricks and PySpark.

• Build ETL/ELT processes for structured and unstructured data

• Work with Delta Lake and Databricks workflows/jobs.

• Develop and optimize SQL queries and data transformations.

• Integrate Databricks with Azure services such as Azure Data Lake Storage (ADLS) and Azure Data Factory.

• Perform data quality checks, troubleshooting, and performance tuning.

• Collaborate with data engineers, analysts, and business teams.

• Follow best practices for security, monitoring, and deployment.

Required Skills

• Hands-on experience with Azure Databricks.

• Strong knowledge of PySpark and Python.

• Good knowledge of SQL.

• Experience with Delta Lake.

• Experience with Azure Data Lake Storage (ADLS).

• Understanding of Azure Data Factory (ADF).

• Knowledge of data warehousing and ETL concepts.

• Good problem-solving and communication skills.

Certification

• Databricks Certified certification is required/preferred.

• Relevant certifications such as Databricks Certified Data Engineer Associate/Professional are an advantage

Key Skills

Education

Bachelor's degree