Description

Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience

Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql

Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql

Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows

Must have experience and a good understanding on the below topics

    Databricks delta lake storage

    Unity catalog

     Autoloader

     Lakehouse architecture

    Medallion architecture

    Databricks serverless options

    Dataset and data frame

    Databricks clusters configuration and sizing

    Databricks job orchestration

    Spark structured streaming

Must have experience with complex and large volume data streaming and data transformations using Azure Databricks

Must have extensive experience in datawarehouse / data lakehouse implementation

Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs

Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services

The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development woek

Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous

Key Skills

Education

Not specified