Description

You will build and automate data pipelines to support machine learning models and feature lifecycles.

Responsibilities

• Design and build batch and incremental data pipelines using Spark SQL and Python orchestrated via Azure Data Factory.
• Implement FeatureOps to manage the discovery, validation, and serving of features for training and inference.
• Collaborate with data scientists to engineer features and prepare accurate training datasets.
• Automate pipelines using CI/CD approaches, including automated testing, deployment, and monitoring.
• Implement data quality frameworks to identify and resolve inconsistencies or anomalies, and optimize pipelines for performance.

Required Skills

• 5+ years of experience as a Data Engineer or in a similar role.
• Proficiency in Python and Java.
• Experience with Spark, Hadoop, SQL, and NoSQL databases.
• Hands-on experience with cloud platforms such as Azure or AWS.
• Practical experience supporting machine learning projects and workflows.
• Familiarity with TensorFlow or PyTorch.
• Knowledge of machine learning concepts.
• Any Graduate degree.

Key Skills

Education

Any Graduate