Description

You will lead a cross-functional Agile team to build and enhance a data ingestion and entity resolution platform.

Responsibilities

• Lead the development of software for data ingestion and entity resolution.
• Architect application, data, and infrastructure solutions for large, complex datasets.
• Build efficient storage and search functions for structured and unstructured data.
• Develop rapid user-driven prototypes to evaluate technical and architectural approaches.
• Design and implement batch and real-time data pipelines for production systems.

Required Skills

• 4+ years of management experience in large-scale data processing and instrumentation.
• Strong coding proficiency in Java, Python, or Scala.
• Experience with big data ecosystems including Spark, Kafka, Hadoop, and Airflow.
• Hands-on experience with Databricks, Delta Lake, Elasticsearch, or Snowflake.
• Proficiency with relational and NoSQL databases such as DB2 and MongoDB.
• Working knowledge of data formats including Parquet, CSV, JSON, XML, and relational data.
• Experience with Iceberg, data modeling, and columnar databases.
• Familiarity with distributed systems, REST-based applications, and object-oriented or functional programming.
• Solid understanding of engineering best practices, design patterns, and code quality.

Preferred Skills

• Experience with the Quantexa platform for entity resolution.
• Experience sourcing client reference data from vendors.

Key Skills

Education

Any Gradute