Description

You will lead the development of scalable data architectures and pipelines specifically designed to support machine learning workflows.

Responsibilities

• Develop and maintain scalable data pipelines to support machine learning models and analytics.
• Design machine learning solutions by integrating new data sources and optimizing data usage.
• Manage complex technical discussions with internal teams and external data partners.
• Ensure data quality, integrity, and security across all pipeline stages.
• Implement monitoring and alerting systems to detect anomalies in processing and model performance.
• Improve data versioning, lineage, and reproducibility for model transparency.
• Drive the Kanban process for product maintenance and advocate for tools to reduce technical debt.

Required Skills

• 5+ years of experience in data engineering with a focus on machine learning workflows.
• Proven expertise in designing and maintaining large-scale ETL processes managing 1TB per day.
• Hands-on experience with Kafka.
• Strong proficiency in SQL and NoSQL database management systems.
• Experience with distributed data processing using Hadoop and Spark.
• Experience with cloud platforms such as AWS or GCP.
• Practical knowledge of data wrangling, preprocessing, and feature engineering.
• Understanding of data governance, security protocols, and compliance requirements.
• Ability to optimize data flows for model training and real-time inference with latency requirements.

Preferred Skills

• Experience with Snowflake and Databricks.
• Background in video-game development, the advertising industry, or automated online transaction businesses.

Key Skills

Education

Any Graduate