Description

Responsibilities

• Design, code, and deploy batch and streaming data pipelines using PySpark, Spark SQL, and Lakeflow. • Ingest data from Point-of-Sale (POS), e-commerce platforms, loyalty systems, and marketing clouds. • Implement data transformations and business logic within the Medallion Architecture (Bronze, Silver, Gold layers). • Build and optimize Gold dimension tables as the single source of truth. • Define governed business KPIs using Unity Catalog Metric Views. • Implement data quality frameworks and cleansing routines to ensure accurate Customer 360 data. • Monitor, debug, and tune Databricks jobs and Spark clusters for performance and cost-efficiency. • Implement best practices for partitioning, caching, and data layout in Delta Lake. • Manage Databricks environments, clusters, and job deployments using Terraform and AWS DevOps/GitHub Actions. • Champion and implement CI/CD best practices for data pipelines. • Implement data governance features within Databricks Unity Catalog, including data lineage tracking, fine-grained access controls, ABAC policies, tags, and data masking. • Partner with Functional Consultants, Data Scientists, and Analytics Engineers to deliver consumption-ready datasets. • Build and govern Unity Catalog Business Semantics, including Metric Views, Domains, Pages, Glossary Terms, and trusted assets. • Configure and curate AI/BI Genie Spaces and the Genie Ontology to enable conversational analytics for business users. • Enable consumption through AI/BI Dashboards and Databricks One. Required Skills • PySpark • Spark SQL • Lakeflow • Lakeflow Connect • Lakeflow Declarative Pipelines • Terraform • AWS DevOps • GitHub Actions
 

Key Skills

Education

Any Gradute