Description

You will architect and implement large-scale enterprise data solutions, managing complex migrations and big data deployments in Scottsdale, AZ.

Responsibilities

• Lead enterprise-wide initiatives including system integration, data migration, and data warehouse builds.
• Design and implement streaming data pipelines, managing window definitions, late data, and data freshness.
• Develop and optimize ETL/ELT processes using structured streaming and Delta Live Tables.
• Debug, troubleshoot, and resolve complex technical issues within high-performance big data applications.
• Brief technology partners and senior management on the benefits and constraints of proposed technical solutions.

Required Skills

• 10+ years of IT experience focusing on enterprise data architecture and management.
• Expertise in Java and Scala programming within Spark environments.
• Advanced SQL skills including windowing functions, CTEs, and RDBMS schema performance optimization.
• Deep knowledge of Databricks, Delta Lake, and Delta Live Tables.
• Extensive experience with AWS services including S3, Glue, Redshift, and Lambda.
• Proficiency in Conceptual, Logical, and Physical Data Modeling (Relational and Dimensional).
• Hands-on experience with indexing, partitioning strategies, and schema evolution.
• Experience managing CI/CD pipelines using GitLab and monitoring via CloudWatch.
• Understanding of message formats such as Avro and ORC.

Preferred Skills

• Experience with Great Expectations or similar data validation frameworks.
• Familiarity with AWS Data Migration Service.

Key Skills

Education

Any Graduate