Description

You will build and maintain the data infrastructure and pipelines required for Generative AI applications.

Responsibilities

• Design and implement data pipelines for ingesting, pre-processing, and transforming data for model training and inference.
• Build and maintain efficient data storage solutions, including data lakes, warehouses, and databases for large-scale datasets.
• Implement data security and governance policies to ensure privacy and integrity of sensitive data.
• Collaborate with architects and MLOps engineers to translate model requirements into efficient data pipelines.
• Monitor and optimize pipelines for performance, scalability, and cost-effectiveness.

Required Skills

• 8+ years of experience in data engineering, ETL/ELT processes, or data pipeline development.
• Proven experience building and maintaining data pipelines specifically for machine learning projects.
• Proficiency in Python, SQL, and scripting languages such as Bash or Shell.
• Hands-on experience with Data Lakes and Data Storage solutions.
• Strong understanding of data modeling, data quality measures, and security best practices.
• Experience with cloud platforms such as AWS, GCP, or Azure for data storage and processing.
• Knowledge of Generative AI, MLOps, and Model Training workflows.
• Ability to work in the EST time zone and travel to project locations for 2-3 weeks initially, and 3-4 days per month thereafter.

Key Skills

Education

Any Graduate