Description

You will build the ML data infrastructure platform, creating the systems and tools for efficient data preparation, feature engineering, and dataset management.

Responsibilities

• Design and implement scalable data processing pipelines for ML training and validation.
• Build and maintain feature stores supporting both batch and real-time features.
• Develop data quality monitoring, validation, and testing frameworks.
• Create systems for dataset versioning, lineage tracking, and reproducibility.
• Implement automated data documentation and discovery tools.

Required Skills

• 7+ years of software engineering experience, with 3+ years in data infrastructure.
• Strong expertise in GCP data and ML infrastructure, including BigQuery, Dataflow, Cloud Storage, Vertex AI Feature Store, Cloud Composer, and Dataproc.
• Deep expertise in data processing frameworks (Spark, Beam, Flink).
• Experience with feature stores (Feast, Tecton) and data versioning tools.
• Proficiency in Python and SQL.
• Experience with data quality and testing frameworks.
• Knowledge of data pipeline orchestration (Airflow, Dagster).
• Experience with Bigquery, Spark, and Data Warehousing concepts.

Key Skills

Education

Any Gradute