Description

You will design, build, and operate scalable batch and stream data processing infrastructure for ML operations, covering training, serving, evaluation, and experimental systems.

Responsibilities

• Design and develop foundational data models, data warehouse, and processing pipelines using technologies like AWS EMR Spark, Apache Kafka, AWS Athena, Snowflake, Airflow, and Apache HUDI.
• Collaborate with ML and Data Science teams to understand data requirements, influence roadmap, and execute on data projects.
• Build and operate a data governance platform covering data cataloging, lineage, audit, deletion, and masking.
• Develop and run an orchestration platform utilizing Temporal and Airflow for other team workflows.
• Build data services and APIs to expose data to stakeholders and customer-facing products.

Required Skills

• 5+ years of experience as a Senior Software Engineer.
• Experience with Python, Golang, Java, or C++.
• Experience with cloud infrastructure such as AWS, GCP, or Azure.
• Experience with relational or non-relational databases, including Postgres, AWS DataLake/S3, DynamoDB, or Snowflake.
• Proven ability to build and operate large-scale data processing systems.

Key Skills

Education

Any Graduate