Description

You will build and scale data ingestion frameworks and pipelines to move structured and unstructured data into the OneData platform.

Responsibilities

• Build data ingestion frameworks to process data from SharePoint, Confluence, Chat Bots, Jira, and external sites.
• Design scalable target state architectures for document-based processing involving XML, HTML, DOC, PDF, XLS, JPEG, TIFF, and PPT.
• Implement vector databases to store and retrieve high-dimensional vectors for RAG architectures and LLMs.
• Automate data quality, security, and validation checks to ensure the integrity and maintainability of solutions.
• Monitor and troubleshoot pipeline performance to resolve bottlenecks and implement incremental ingestion strategies.

Required Skills

• 10+ years of experience in database and cloud-based technologies.
• Proficiency in Python, SQL, and PySpark.
• Experience with Databricks and Snowflake.
• Hands-on expertise with CI/CD tools including Jenkins, Terraform, Splunk, and Dynatrace.
• Experience with Machine Learning and AI concepts, specifically RAG architecture, LLMs, and embeddings.
• Experience working with Cloud data stores, NoSQL, Graph, and Vector databases.
• Knowledge of structured and unstructured data formats including XML, HTML, DOC, PDF, XLS, JPEG, and TIFF.
• Proficiency with GitHub, Jira, and Confluence.
• Bachelor’s degree in Engineering, Computer Science, or a related field.

Key Skills

Education

Bachelor’s Degree