← Back to jobs

Avance Consulting Logo
Data Engineer

Avance Consulting

 

Quincy, IL, USA

Posted On: Just posted
Experience: 8+ years
Availability: Onsite
Openings: 1
Category: Data Engineer
Tenure: No Preference/Any
Related Jobs

No related jobs found

Description

Must Have Technical/Functional Skills

  • Must have at least 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines.
  • SQL/Procedure experience.
  • Must have 8 years of hands-on development proficiency in implementing Databricks solutions using Scala and Spark with data frames and notebooks/SQL.
  • Must have 8+ years of hands-on development and performance tuning/enhancement experience in Scala/PySpark and SQL.
  • Must have strong experience with Databricks, including developing and optimizing Spark jobs, data transformations, and data processing workflows.
  • Must have experience and a good understanding of the above topics.
  • Databricks Delta Lake storage
  • Unity Catalog
  • Autoloader
  • Lakehouse architecture
  • Medallion architecture
  • Databricks serverless options
  • Dataset and DataFrame
  • Databricks clusters configuration and sizing
  • Databricks job orchestration
  • Spark structured streaming

Roles & Responsibilities

  • Must have experience with complex and large volume data streaming and data transformations using Azure Databricks.
  • Must have extensive experience in data warehouse / data lakehouse implementation.
  • Must have experience with monitoring, debugging, and resolving issues in live Databricks jobs.
  • Must have strong hands-on expertise in troubleshooting DevOps pipelines, Azure, and AKS services.
  • The candidate should be able to provide data ingestion, streaming, and transformations solutions and do development work.
  • Experience with Apache big data implementation with programming proficiency like PySpark, R, or Java prior to Databricks is advantageous.
  • Good to have experience with containerization technologies such as Docker and Kubernetes

Education

Any Graduate

Related Jobs

No related jobs found

← Back to jobs