Description

You will architect and scale intelligent data systems focused on code-based text datasets to support AI and ML pipelines.

Responsibilities

• Design and implement scalable data infrastructure to ingest, transform, and manage large-scale code datasets.
• Build systems for automated web crawling, parsing, deduplication, and metadata extraction from code repositories.
• Develop robust data pipelines for processing structured text datasets using distributed compute frameworks.
• Monitor data quality, throughput, and overall system performance.
• Collaborate with research, infrastructure, and compliance teams to meet technical requirements.

Required Skills

• 5+ years of software engineering experience in data-intensive environments.
• Proven experience building and maintaining scalable data systems and infrastructure.
• Experience with web crawling and large-scale data ingest.
• Comfortable with AWS, including storage, containerized compute, and security.
• Working experience with Python, Go, or Scala.
• Experience with Spark or Ray, Kafka, PostgreSQL, and GitHub APIs.
• Understanding of how datasets feed into AI/ML workflows.

Key Skills

Education

Any Graduate