Most agent benchmarks are toys. A grid world, a simplified room, a game. This one is Hong Kong, replicated from territory-wide 3D geospatial data, with first-person navigation and spatial reasoning tasks.
The question it asks is the one that matters for anything autonomous on a site or in a building: can an agent turn what it sees locally into dependable action across a real, large, messy space?
The answer, and it is a useful no
Contemporary multimodal agents showed useful atomic abilities in visual recognition. They can look at a thing and say what it is. But they struggled with sustained goal-directed urban exploration, and the paper is precise about the failure mode: errors accumulate without effective correction.
It is not that the agent cannot see. It is that it cannot notice it has been wrong for the last twenty minutes.
- 1
Perceive
Recognise objects and features in the immediate view. Reported as a genuine strength of current models.
- 2
Decide
Choose the next action from what is visible now. Also broadly works.
- 3
Sustain
Hold the goal across many steps while the scene changes. This is where performance falls away.
- 4
Correct
Notice that the accumulated path has drifted and recover. The reported failure point.
Why a negative result is the useful one this fortnight
Four of the five papers I read this fortnight report improvements. This one reports a boundary, and boundaries are what you need when someone is selling you autonomy.
The pattern generalises well beyond city navigation. Any agent operating over a long horizon in a physical environment faces the same structure: perception is solved enough, single decisions are fine, and the difficulty is accumulated error nobody catches. That is exactly the failure mode a site robot, an autonomous survey drone or a long-running inspection agent will exhibit.
The design implication
- Keep horizons short. Break long autonomous tasks into checkpointed segments where state is verified before the next segment starts.
- Build the correction mechanism explicitly. Do not assume the model will notice its own drift, because this paper says it will not.
- Evaluate at realistic scale. An agent that works in a demo room tells you nothing about one that works across a floor plate, let alone a site.
- Treat perception scores as the easy metric. Recognition accuracy is not evidence of competence at anything sustained.