Description

You will design and maintain scalable data pipelines to ingest, process, and transform large volumes of structured and unstructured data for LLM training and evaluation.

Responsibilities

• Build and implement efficient data processing workflows to support Large Language Model solutions.
• Integrate LLMs into data pipelines for natural language processing tasks.
• Deploy, scale, and monitor AI solutions across Snowflake, Azure, and AWS environments.
• Manage cloud costs and optimize resource usage to minimize expenditure.
• Resolve issues regarding data quality, pipeline failures, and resource contention.

Required Skills

• 5+ years of experience in Data Engineering and ETL processes.
• Proficiency in Python and data manipulation tools.
• Hands-on experience with Snowflake, Azure, and AWS.
• Experience with Gen AI workflows and Large Language Models.
• Expertise in handling sensitive data using HIPAA and FHIR standards, including PII and PHI masking techniques.
• Knowledge of search algorithms, indexing, and retrieval models.
• Familiarity with Elasticsearch or Azure AI Search.
• Experience with chunking techniques, vectors, and vector databases like Pinecone.
• Ability to preprocess and clean large datasets efficiently.

Key Skills

Education

Any Graduate