Description

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build document ingestion, classification, extraction, validation, and transformation pipelines for PDFs, emails, attachments, Excel/XLSX files, insurance applications, Statements of Values (SOVs), and other business documents.
• Integrate LLMs and AI models into data-processing workflows to extract structured information from unstructured and semi-structured documents.
• Troubleshoot false positives, incorrect metadata, and inaccurate field values generated by LLM-based extraction pipelines.
• Implement document classification, source-specific extraction rules, data validation, reconciliation, and quality-control mechanisms.
• Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
• Design and implement ontology-based data models representing insurance entities, business concepts, and relationships.
• Map extracted document attributes to standardized ontology concepts and metadata structures.
• Optimize the processing of large and complex Excel/XLSX and SOV files, addressing performance, scalability, and concurrency challenges.
• Implement reliable data persistence using PostgreSQL, Lakebase, or related database technologies.
• Develop and maintain APIs and data products to support downstream application integration.
• Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
• Monitor production pipelines, troubleshoot data-processing issues, and continuously improve reliability, performance, and data quality.

Basic Qualifications: (What are the skills required for this job with minimum years of experience on each)

• Minimum 7+ years of experience in data engineering, software development, or production data pipeline development.
• Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines.
• Minimum 5+ years of experience with Python, PySpark, and SQL.
• Minimum 3+ years of experience processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
• Minimum 2+ years of experience integrating LLMs or AI models into document-processing or data-extraction pipelines.
• Minimum 3+ years of experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
• Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
• Minimum 2+ years of experience designing JSON outputs, APIs, and data products for downstream application consumption.
• Minimum 2+ years of experience with PostgreSQL, Lakebase, or similar relational database technologies.
• Minimum 2+ years of experience troubleshooting production performance, scalability, concurrency, and data-processing issues.
• Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
• Strong analytical, problem-solving, communication, and cross-functional collaboration skills.

Degree:

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equivalent professional experience

Key Skills

Education

Bachelor's degree