Description

You will build and maintain data pipelines to preprocess and mitigate risks in large-scale image, video, and text datasets for research teams.

This role is remote.

Responsibilities

• Preprocess raw datasets into formats consumable by mitigation pipelines.
• Run filtering processes using Integrity's pipeline and consume results to update, repackage, and re-ingest data.
• Implement optimizations to improve data processing workflows and efficiency.
• Perform data quality control and validation on large datasets.

Required Skills

• 5+ years of professional experience in data engineering or related fields.
• Proficiency in Python for scripting file processing and data transfer automation.
• Experience building and managing robust data pipelines.
• Strong skills in data processing, cleaning, transformation, and formatting.
• Experience with SQL and processing large datasets.
• Effective communication for collaboration with stakeholders.

Key Skills

Education

Any Graduate