Description

You will design, build, and test data pipelines using Azure Databricks.

Responsibilities

• Design and build data pipelines leveraging Azure Databricks.
• Test data pipelines to ensure data integrity and quality.
• Perform data analysis using SQL to understand source data.
• Manage clusters and optimize Spark workloads.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency with Python and PySpark.
• Strong SQL skills for data analysis and manipulation.
• Experience with Azure Databricks and cluster management.
• Working knowledge of Apache Spark.
• Experience with Microsoft Azure cloud platforms.
• Familiarity with Azure Synapse Analytics Dedicated SQL Pools (SQLDW).
• Experience working with JSON formats.
• Knowledge of relational and NoSQL databases, specifically CosmosDB.

Preferred Skills

• Deep understanding of MS Azure data engineering technologies.

Key Skills

Education

Any Graduate