Description

Responsibilities:

Design and develop scalable data engineering solutions using Databricks and Apache Spark. Build and optimize ETL/ELT pipelines using PySpark, Python, or Scala. Implement Delta Lake architecture and distributed data processing solutions. Optimize Spark jobs, Databricks clusters, pipelines, performance, and cloud costs. Develop CI/CD, testing, monitoring, and deployment processes for Databricks workloads. Troubleshoot complex data pipeline and Spark performance issues and perform root-cause analysis. Support ML model deployment and lifecycle management within Databricks. Collaborate with data engineers, data scientists, architects, and business stakeholders in Agile environments. Required Skills: Databricks Apache Spark / PySpark Delta Lake Python or Scala ETL/ELT & Data Engineering Azure/AWS/GCP

Key Skills

Education

Bachelor’s degree