Description

You will build and scale the data systems required for machine learning training and validation.

Responsibilities

• Design and implement scalable data processing pipelines for ML workloads.
• Build and maintain feature stores supporting both batch and real-time features.
• Develop frameworks for data quality monitoring, validation, and testing.
• Create systems for dataset versioning, lineage tracking, and reproducibility.
• Partner with data scientists to optimize data preparation workflows.

Required Skills

• 7+ years of software engineering experience, with 3+ years focused on data infrastructure.
• Deep expertise in GCP data and ML infrastructure including BigQuery, Dataflow, Cloud Storage, Vertex AI Feature Store, Cloud Composer, and Dataproc.
• Proficiency in Python and SQL.
• Expertise in data processing frameworks such as Spark, Beam, or Flink.
• Experience with feature stores like Feast or Tecton and data versioning tools.
• Experience with data pipeline orchestration using Airflow or Dagster.
• Experience with data quality and testing frameworks.

Preferred Skills

• Experience with streaming systems such as Kafka, Kinesis, or Pub/Sub.
• Knowledge of GCP security, IAM, Cloud Logging, Cloud Monitoring, Cloud Build, or Cloud Deploy.
• Familiarity with ML metadata management, data governance, or dbt.

Key Skills

Education

Any Graduate