Description

Key Responsibilities

Solution Architecture & Design

• Design and architect scalable, secure, and high-performance data platforms using Azure Databricks.
• Lead the design and implementation of modern data lakehouse architectures utilizing Delta Lake, Unity Catalog, and Databricks best practices.
• Define data ingestion, transformation, orchestration, governance, and consumption patterns across enterprise data ecosystems.
• Establish architecture standards, development frameworks, and reusable components.

Data Engineering & Development

• Develop and optimize complex data pipelines using PySpark and Databricks.
• Build and maintain ETL/ELT frameworks leveraging Delta Live Tables (DLT) and Databricks Workflows.
• Ensure code quality, performance optimization, scalability, and reliability of data processing solutions.
• Implement data quality, monitoring, lineage, and governance controls.

Project Delivery

• Lead end-to-end Databricks implementations from strategy and design through deployment and stabilization.
• Collaborate with business stakeholders, data consumers, architects, and platform teams to translate business requirements into technical solutions.
• Guide development teams and perform design/code reviews.
• Troubleshoot complex technical challenges and provide solution recommendations.

Client Engagement & Consulting

• Serve as a trusted advisor to client stakeholders on data modernization and analytics transformation initiatives.
• Conduct architecture workshops, solution demonstrations, and technical assessments.
• Present technical solutions and recommendations to business and executive audiences.
• Support estimation, solutioning, proposal development, and pre-sales activities as needed.

Required Qualifications

• 10+ years of overall experience in Data Engineering, Data Architecture, or Analytics platform delivery.
• 5+ years of hands-on experience with Databricks.
• Demonstrated experience delivering at least two (2) end-to-end Databricks implementations in enterprise environments.
• Strong hands-on coding expertise in PySpark with the ability to develop solutions during technical interviews and coding assessments.
• Extensive experience with:
  • Unity Catalog
  • Delta Lake
  • Delta Live Tables (DLT)
  • Databricks Workflows
• Strong experience designing and implementing Azure Databricks architectures.
• Experience integrating Databricks with Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse, Azure Key Vault, and Microsoft Fabric (preferred).
• Strong understanding of data modeling, ETL/ELT design patterns, data governance, and performance tuning.
• Proven client-facing consulting experience with excellent communication and stakeholder management skills.

Preferred Qualifications

• Databricks Certified Professional or Associate certifications.
• Azure Data Engineering and/or Azure Solutions Architect certifications.
• Experience implementing CI/CD, Infrastructure as Code, and DevOps practices for Databricks environments.
• Experience with data governance, metadata management, security, and access controls in enterprise data platforms.
• Familiarity with streaming architectures, real-time analytics, and advanced analytics workloads.

Key Competencies

• Enterprise Data Architecture
• Data Lakehouse Design
• PySpark Development
• Azure Databricks
• Solution Leadership
• Client Relationship Management
• Technical Consulting
• Data Governance & Security
• Problem Solving & Analytical Thinking
• Team Leadership & Mentoring

Key Skills

Education

Bachelor's degree