Description

Key Responsibilities

Design and architect enterprise-scale Databricks Lakehouse solutions.

Define modern data architecture strategies leveraging Databricks, Apache Spark, Delta Lake, and cloud platforms.

Lead the design and implementation of scalable data ingestion, transformation, processing, and analytics solutions.

Develop and optimize large-scale Spark workloads using PySpark/Scala and SQL.

Design data pipelines and workflows using Databricks and cloud-native services.

Establish best practices for Lakehouse architecture, data modeling, security, governance, and performance optimization.

Work closely with data engineers, data scientists, architects, and business stakeholders to translate requirements into technical solutions.

Design solutions for batch and near-real-time/streaming data processing.

Implement and optimize Delta Lake capabilities, including ACID transactions, schema evolution, partitioning, and data optimization.

Drive cloud data platform architecture across AWS, Azure, and/or GCP.

Provide technical leadership and mentorship to data engineering teams.

Evaluate emerging technologies and recommend solutions aligned with enterprise data strategies.

Troubleshoot complex performance, scalability, reliability, and data quality issues.

Required Skills

15+ years of experience in data engineering, data architecture, or related technology roles.

Strong hands-on experience with Databricks and Lakehouse Architecture.

Expert-level knowledge of Apache Spark, including PySpark and/or Scala.

Strong experience with Delta Lake and modern data lake/lakehouse concepts.

Extensive experience with at least one major cloud platform:

Microsoft Azure

AWS

Google Cloud Platform (GCP)

Strong SQL and data modeling skills.

Experience designing enterprise-scale data pipelines and ETL/ELT frameworks.

Strong understanding of distributed computing and big-data architecture.

Experience with cloud data services, storage, compute, networking, and security.

Knowledge of data governance, security, lineage, metadata management, and access controls.

Experience with CI/CD, DevOps, Git, and infrastructure/application deployment practices.

Excellent architecture, problem-solving, communication, and stakeholder-management skills.

Preferred Qualifications

Databricks certifications such as Databricks Certified Data Engineer, Data Engineer Professional, or related certifications.

Experience with Unity Catalog and Databricks governance capabilities.

Experience with streaming technologies such as Kafka, Structured Streaming, or equivalent.

Experience with orchestration tools such as Databricks Workflows, Airflow, Azure Data Factory, or similar.

Experience integrating Databricks with enterprise BI and analytics platforms.

Experience implementing data mesh, data governance, or enterprise data modernization initiatives

Key Skills

Education

Bachelor's degree