Description

You will build automated data pipelines, manage large-scale data processing, and integrate AI-driven components into ingestion workflows.

This role is on-site.

Responsibilities

• Build automated data pipelines using Airflow, Luigi, or custom schedulers for extraction and transformation tasks.
• Develop web data collection tools handling pagination, AJAX, JavaScript-rendered content, and rate-limiting.
• Integrate and manage APIs for data ingestion with robust error handling and logging.
• Implement AI/ML workflows using PyTorch, TensorFlow, LangChain, LangGraph, ChainLit, and Vertex AI.
• Store and query data using PostgreSQL, MongoDB, and cloud storage solutions like AWS S3 or Google Cloud Storage.

Required Skills

• 8+ years of professional experience with Python for data engineering and automation.
• Strong proficiency with Apache Spark for large-scale data processing.
• Hands-on experience with machine learning frameworks: PyTorch and TensorFlow.
• Experience building agents or workflows with LangChain, LangGraph, and ChainLit.
• Practical knowledge of Google Cloud Vertex AI.
• Solid understanding of RESTful APIs and API-based data ingestion patterns.
• Experience with relational (PostgreSQL) and NoSQL (MongoDB) databases.
• Proficiency with Git for version control and collaborative development.
• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree