Description

You will engineer and maintain the data infrastructure supporting machine learning models.

Responsibilities

• Build and manage ETL pipelines for real-time and batch model serving in production.
• Develop and optimize data preparation processes, focusing on ML feature engineering.
• Implement data workflows utilizing the medallion architecture on Databricks.
• Ensure data security and compliance, particularly when handling sensitive data like PII.

Required Skills

• 7+ years in data engineering roles.
• 4+ years focused specifically on ML feature engineering and ETL pipeline development.
• Expert proficiency in Python for data manipulation.
• Strong command of Apache Spark and PySpark for big data processing.
• Proficiency in SQL for query optimization and data warehousing.
• Experience building and managing pipelines on Databricks.
• Familiarity with CI/CD practices for ETL/ELT pipeline development.
• Experience in metadata and master data management within technical data catalogues.

Key Skills

Education

Any Graduate