Description

You will design, build, and operate scalable batch and stream data processing infrastructure for ML operations, covering training, serving, evaluation, and experimental systems.

This role is on-site.

Responsibilities

• Design and develop foundational data models, data warehouses, and processing pipelines using AWS EMR Spark, Apache Kafka, AWS Athena, Snowflake, Airflow, and Apache HUDI.
• Build and operate a data governance platform covering data cataloging, lineage, audit, deletion, and masking.
• Develop and run an orchestration platform utilizing Temporal and Airflow for other team workflows.
• Build data services and APIs to expose data to stakeholders and customer-facing products.
• Collaborate with ML and Data Science teams to understand data requirements, influence roadmap, and execute on data projects.

Required Skills

• 5+ years of experience as a Senior Software Engineer.
• Proficiency in Python, Golang, Java, or C++.
• Experience with cloud infrastructure such as AWS, GCP, or Azure.
• Experience with relational or non-relational databases, including Postgres, AWS DataLake/S3, DynamoDB, or Snowflake.
• Proven ability to build and operate large-scale data processing systems.

Preferred Skills

• Experience with Temporal for workflow orchestration.
• Familiarity with ML operations lifecycles.

Key Skills

Education

Any Graduate