Description

You will lead and architect enterprise-wide data initiatives including system integration, data migration, and the implementation of data lakes and warehouses.

Responsibilities

• Architect enterprise-scale data migrations, transformations, and data mart builds.
• Design and implement complex streaming data pipelines and batch systems.
• Debug, troubleshoot, and resolve complex technical issues within big data applications.
• Brief stakeholders and senior management on the benefits and constraints of proposed technology solutions.
• Maintain GitLab CI/CD pipelines and monitor systems using CloudWatch.

Required Skills

• 8+ years of IT experience with a focus on enterprise data architecture and management.
• Expertise in Conceptual, Logical, and Physical Data Modelling, including Relational and Dimensional models.
• Hands-on experience with Databricks, Spark Scala, Structured Streaming, Delta Lake, and Delta Live Tables.
• Advanced SQL skills including windowing functions, CTEs, joins, and RDBMS schema performance optimization.
• Deep understanding of ETL/ELT processes and incremental data load strategies like tumbling/sliding windows and high watermarks.
• Experience managing data lake concepts such as time travel, schema evolution, and optimization.
• Proficiency with AWS services including S3 and Lambda for data processing and configuration.
• Experience with indexing, partitioning strategies, and large-scale big data application deployment.
• Knowledge of Schema Registry and message formats such as Avro and ORC.
• Technical skills in Spark, S3, Glue, AWS Redshift, and Python.

Preferred Skills

• Experience with Great Expectations or similar data quality frameworks.
• AWS architecture experience and familiarity with StreamSets.

Key Skills

Education

Any graduate