What's hard about web-navigation agents?

hard

Answer

$(1) Grounding: mapping the DOM / screenshot to actionable elements is fragile — CSS selectors break, elements are dynamic, invisible overlays.$
$(2) Latency: page loads add seconds per step.$
$(3) Auth / captchas: most sites need cookies, MFA, or human verification.$
$(4) State: sessions expire mid-agent.$
$(5) Cost: full-page screenshots are expensive input tokens.$
$Benchmarks: WebArena, VisualWebArena, Mind2Web.$
$Successful agents (Anthropic Computer Use, browser-use, OpenAI Operator) combine vision + accessibility trees + long-context reasoning.$

Check yourself — multiple choice

Trivial task
Fragile DOM grounding + latency + auth/captchas + session state + expensive screenshots — combine vision + a11y trees + long-context
Only slow
Same as chat

Web agents: brittle grounding + auth + state + cost; combine vision + a11y + long-context.

#agents#multimodal

214 interview questions in this topic.