Description

You will build and manage data pipelines within the Azure ecosystem using Databricks and Spark.

Responsibilities

• Create and configure Azure Databricks services to support data engineering workflows.
• Build ingestion, transformation, and egress pipelines using Azure Data Factory.
• Process data in various formats, including CSV, Parquet, and JSON, sourced from Snowflake DWH.
• Develop application services using PySpark and Spark SQL.

Required Skills

• 4+ years of professional data engineering experience.
• Hands-on experience with Azure Databricks services and Databricks File System (DBFS).
• Proficiency in PySpark and Spark SQL.
• Experience working with RDD and Dataframe APIs.
• Practical knowledge of Azure Data Factory for pipeline construction.
• Experience managing data movement between Snowflake and Azure storage layers.
• Working knowledge of Azure SQL database.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree