Description

You will build and maintain automated data pipelines and ingestion systems. Your work focuses on extracting data, processing it at scale, and integrating AI-driven components into the workflow.

Responsibilities

• Build automated data pipelines using Airflow, Luigi, or custom schedulers to handle extraction and transformation tasks.
• Develop web data collection tools that handle pagination, AJAX, JavaScript-rendered content, and rate-limiting.
• Integrate and manage APIs for data ingestion, ensuring robust error handling and logging.
• Implement AI/ML workflows using PyTorch, TensorFlow, LangChain, LangGraph, ChainLit, and Vertex AI.
• Store and query data using PostgreSQL, MongoDB, and cloud storage solutions like AWS S3 or Google Cloud Storage.

Required Skills

• 8+ years of professional experience with Python for data engineering and automation.
• Strong proficiency with Apache Spark for large-scale data processing.
• Hands-on experience with machine learning frameworks: PyTorch and TensorFlow.
• Experience building agents or workflows with LangChain, LangGraph, and ChainLit.
• Practical knowledge of Google Cloud Vertex AI.
• Solid understanding of RESTful APIs and API-based data ingestion patterns.
• Experience with relational (PostgreSQL) and NoSQL (MongoDB) databases.
• Proficiency with Git for version control and collaborative development.
• Bachelor's degree in Computer Science or related field.

Preferred Skills

• Experience with AWS services beyond S3, such as EC2, Lambda, or Glue.
• Ability to write clean, modular, and well-documented code with strong debugging skills.

Key Skills

Education

Bachelor's degree