MLOps

Experiments -> Champion new experiments -> Challenger

Challenger can de-throne the champion model

• feature drift
  • Are the patterns in the features changing ?
• Inference drift
• Quantitative vs qualitative drifts
  • example: predict price and show on website. But everyone things its high price and no bookings. No way to say there is drift. So we rely on business metrics, so we can say that no one has booked, like booking conversion rate. Or even surveys.
• Feature stores
  • if data is produced from multiple sources, and during training, we do some transformations. Ex: Col A -> Col A'
  • Hence, during inference, we want to transform the incoming data in the same way, so we provide the production model with same transformed A'.
  • This is where a feature store will help, which helps keep transformations centralised and consistent.