Description

Key Responsibilities

Design and implement scalable, secure, and high-performance data architectures using Databricks.

Architect and optimize Lakehouse solutions using Delta Lake, Delta Live Tables/Lakeflow, and Unity Catalog.

Design and manage batch and real-time data pipelines using Apache Spark and PySpark.

Develop and optimize ETL/ELT pipelines for structured and unstructured data.

Implement data lakehouse architecture, data modeling, data integration, and data transformation strategies.

Work with cloud platforms such as AWS, Azure, or GCP to build cloud-native data solutions.

Establish data governance, data quality, security, access control, and metadata management standards.

Optimize Spark workloads, SQL queries, cluster performance, and overall platform costs.

Collaborate with data engineers, business stakeholders, and solution architects to translate business requirements into technical solutions.

Define architecture best practices, reusable frameworks, and technical standards.

Required Skills

Strong hands-on experience with Databricks Lakehouse Platform.

Expertise in Apache Spark, PySpark, Spark SQL, and Python.

Strong knowledge of Delta Lake, Unity Catalog, and Lakehouse architecture.

Experience designing enterprise-scale ETL/ELT pipelines and data integration solutions.

Proficiency in SQL, data modeling, dimensional modeling, and database design.

Experience with at least one major cloud platform: AWS, Microsoft Azure, or GCP.

Knowledge of data governance, data security, data quality, and metadata management.

Experience with performance tuning, distributed data processing, and scalable data solutions.

Strong communication, analytical, and problem-solving skills

Key Skills

Education

Bachelor's degree