Description

Design and implement scalable data solutions on Databricks for engineering, machine learning, and real-time analytics.

This role is remote.

Responsibilities

• Architect and build ETL pipelines using Databricks, Apache Spark, and Delta Lake.
• Optimize Databricks clusters for performance, reliability, and cloud resource efficiency.
• Implement data governance, security, and compliance standards on the platform.
• Mentor junior engineers and define best practices for pipeline development.
• Integrate machine learning workflows and automate model deployment.

Required Skills

• 5+ years of experience in data architecture and engineering with AWS, Azure, or GCP.
• Extensive hands-on experience with Databricks and Apache Spark.
• Proficiency in Python, Scala, or SQL.
• Deep knowledge of Delta Lake, Spark SQL, and MLlib.
• Expertise deploying workloads on AWS, Azure, or GCP.
• Experience with cloud-native services (S3, Redshift, Azure Blob, BigQuery).
• Strong understanding of data modeling, distributed computing, and OLAP/OLTP systems.

Preferred Skills

• Experience with DevOps, automation, and CI/CD pipelines in data engineering.
• Databricks Certified Associate Developer for Apache Spark.
• Cloud certifications (AWS Certified Solutions Architect or Azure Solutions Architect Expert).

Key Skills

Education

Bachelor's Degree