Description

RESPONSIBILITIES

• Design, code, and deploy batch and streaming data pipelines using PySpark, Spark SQL, and Lakeflow to ingest data from POS, e-commerce platforms, loyalty systems, and marketing clouds. • Implement data transformations and business logic within the Medallion Architecture (Bronze, Silver, Gold layers). • Build and optimize Gold dimension tables as the single source of truth. • Define governed business KPIs using Unity Catalog Metric Views. • Implement data quality frameworks and cleansing routines to ensure accurate and trustworthy Customer 360 data. • Monitor, debug, and optimize Databricks jobs and Spark clusters for performance and cost efficiency. • Implement partitioning, caching, and Delta Lake data layout best practices. • Manage Databricks environments, clusters, and deployments using Terraform and AWS DevOps/GitHub Actions. • Implement CI/CD best practices for data pipelines. • Implement Unity Catalog governance capabilities, including data lineage, access controls, tags, and data masking. • Partner closely with Functional Consultants, Data Scientists, and Analytics Engineers to deliver consumption-ready datasets. • Build and govern Unity Catalog Business Semantics, including Metric Views, Domains, Pages, Glossary terms, and certified assets. • Configure and curate AI/BI Genie Spaces and the Genie Ontology to enable conversational analytics and AI-driven data consumption

Key Skills

Education

Bachelor's degree