Description

You will build and scale a hosted, multi-cluster data warehouse for Web3 environments.

Responsibilities

• Write Scala code using Apache Spark and Apache Arrow to develop a multi-cluster data warehouse.
• Develop database optimizers, query planners, data routing mechanisms, and workload management techniques.
• Scale infrastructure and architecture from proof of concept to hundreds of clusters managing hundreds of terabytes.
• Codify engineering patterns, templates, and code bases to facilitate metadata management.
• Collaborate with software engineers to build an optimized HTAP database using open source data tools.

Required Skills

• 6+ years of experience engineering software and enterprise-scale data warehouses or data platforms.
• 3+ years of experience with Scala and Apache Spark.
• Proficiency with Apache Kafka.
• Experience with the Apache stack, specifically Apache Spark and Apache Arrow.
• Strong scripting and test automation knowledge.
• Experience working within rapid development cycles in web-based environments.
• Bachelor's degree.

Preferred Skills

• Experience with query planning, optimization, and distributed data warehouse systems.
• Knowledge of blockchain indexing, Web3 compute paradigms, proofs, and consensus mechanisms.
• Experience with Apache Ignite or Geode.

Key Skills

Education

Bachelor's degree