How does shadow evaluation work?

medium

Answer

$New model runs on production requests in parallel with current model.$
$Predictions logged, not served.$
$Compute: (1) agreement rate with production model.$
$(2) predicted-metric distribution vs production.$
$(3) latency + memory under real load.$
$(4) if labels available (delayed), online accuracy.$
$Purpose: safety check before A/B — catch latency issues, prediction anomalies, correlations with production.$
$No user impact.$
$Typical duration: 1-7 days.$
$Standard pre-A/B step.$

Check yourself — multiple choice

Random
New model runs on prod requests parallel to current, predictions logged not served; compute agreement + distribution + latency/mem under load + delayed accuracy; safety check pre-A/B; no user impact; 1-7 days
Same as A/B
Not real

Shadow eval: parallel run without serving; safety check pre-A/B.

#deployment#monitoring

215 interview questions in this topic.