Description

Role Summary

• We are seeking a Databricks Engineer to lead the design and implementation of a scalable Sales Data Platform as part of the One Data initiative on Databricks running on AWS.
• The role is hands-on and architecture-driven, focused on data ingestion, transformation, modelling, and optimization using modern Lakehouse patterns.
• The architect will work closely with data engineers, source system teams, and downstream consumers to deliver high-quality, governed, and performance-optimized sales datasets.

Key Responsibilities
Architecture & Design

• Define end-to-end Lakehouse architecture on Databricks (AWS) for Sales data domains.
• Design medallion architecture (Bronze / Silver / Gold) aligned with One Data standards.
• Establish data modelling standards for Sales facts, dimensions, hierarchies, and aggregations.
• Define scalable ingestion patterns for batch and incremental loads.
• Drive performance, scalability, and cost optimization best practices.

Data Engineering & Implementation

• Build and guide development of PySpark-based data pipelines in Databricks.
• Implement Delta Lake features:
  • ACID transactions.
  • Schema evolution & enforcement.
  • Time travel & versioning.
• Design and optimize large-scale joins, aggregations, and window functions.
• Implement CDC and incremental processing using watermarking and change detection.
• Ensure idempotent, restart able, and fault-tolerant pipelines.

AWS & Platform Integration

• Architect solutions using AWS services:
  • Amazon S3 (data lake storage).
  • IAM (security & access control).
  • AWS Glue / Glue CatLog.
  • CloudWatch (monitoring & logging).
• Optimize Databricks cluster configurations (job vs all-purpose clusters).
• Implement secrets management and secure connectivity patterns.

Data Quality, Governance & Reliability

• Define and implement data quality checks and validations.
• Ensure data lineage and metadata capture.
• Implement error handling, auditing, and reconciliation frameworks.
• Support data governance and access control requirements.

DevOps & Operational Excellence

• Implement CI/CD pipelines for Databricks notebooks and jobs.
• Enforce code versioning, reviews, and deployment standards.
• Design monitoring, alerting, and SLA tracking for pipelines.
• Support production stabilization and performance tuning.

Collaboration & Leadership

• Act as technical lead / mentor for Databricks data engineers.
• Collaborate with:
  • Source system teams (Sales, CRM, ERP).
  • Data consumers (analytics, downstream apps).
  • Cloud/platform teams.
• Translate business requirements into robust technical designs.

Required Skills & Experience
Core Technical Skills

• 8+ years in Data Engineering / Data Architecture roles.
• 4+ years hands-on experience with Databricks.
• Strong expertise in PySpark & Spark SQL.
• Strong expertise in dbt.
• Deep experience with Delta Lake.
• Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch).

Data Engineering Expertise

• Sales data domain experience (orders, revenue, pricing, customers, products).
• Strong understanding of:
  • Fact & dimension modelling.
  • Slowly Changing Dimensions (SCD Type 1 / 2).
  • Large-scale data processing patterns.
• Experience handling high-volume, high-velocity datasets.

Platform & Operational Skills

• Databricks job orchestration and scheduling.
• Cluster sizing and performance tuning.
• CI/CD for data platforms.
• Strong troubleshooting and debugging skills.

Nice-to-Have

• Experience with enterprise One Data / Data Mesh programs.
• Exposure to real-time or near-real-time ingestion patterns.
• Experience integrating CRM / Sales systems (e.g., Salesforce, SAP Sales data).
• AWS certifications or Databricks certifications

Key Skills

Education

Any Graduate