Description

You build scalable crawler, ingestion, enrichment, and indexing pipelines that power AI Search services.

Responsibilities

• Develop and evolve a modular, JavaScript/TypeScript-based crawler framework for large-scale data extraction.
• Configure new crawler jobs by collaborating with data source owners and service teams.
• Integrate with REST and GraphQL APIs to acquire and enrich data.
• Ingest and process datasets using AI/ML in Databricks with PySpark and Delta Lake into target systems like OpenSearch.
• Troubleshoot crawler jobs on Kubernetes, analyzing logs and debugging issues.

Required Skills

• 5+ years of professional working experience in software development.
• Strong proficiency in Node.js, TypeScript, and Python/PySpark.
• Proven experience with web crawling, HTML parsing, and REST/GraphQL API integration.
• Hands-on experience with Databricks, Delta Lake, and Azure ADLS Gen2.
• Experience understanding and utilizing Kubernetes and Docker for containerized execution.
• Experience designing, implementing, and maintaining CI/CD pipelines.
• Strong log analysis and troubleshooting capabilities.
• Proficiency in German and English.

Key Skills

Education

Any Graduate