Benchmarking how LLMs use location tools

The problem isn't what the model knows, it's what it can't compute

Ask a frontier model how to get somewhere, how long it'll take, or whether your specific truck can even make the trip and it gives you a convincing answer. These are exactly the kinds of questions large language models (LLMs) are built to answer. But when the question depends on the physical world, fluency isn't verification.

In an earlier blog post, we looked at why answering these questions takes more than fluent language. It requires spatial computation. This post puts that argument to a test.

Here is what that looks like. We asked a frontier model, with no location tools attached for the fastest tractor-trailer route from Manhattan to Montauk. It answered with confidence and a long list of named roads. Among its options for leaving Manhattan was the FDR Drive, which doesn't allow commercial vehicles. It called its route the fastest without giving a distance or a travel time, and with nothing to check either against:

The model isn’t at fault here. Without location tools, it has nothing to check its answer against. The fix is simple to state but harder to build well: give the model real location tools to reason with. That’s what the rest of this blog post will show.

Web search fixes stale facts, not spatial reasoning

At first, web search seems like the obvious answer. If the model doesn't know something about a place, just let it look it up. That works when the missing information is a fact. It doesn't work when the answer depends on where things are, how you get between them or what the journey looks like right now.

Search can tell you that a cafe exists, what people think of it and when it opens. But it can't reliably tell you that getting there means crossing six lanes with no pedestrian crossing or that a trip that takes 15 minutes at 2pm could take 40 minutes at 6pm on a Friday. Those aren't facts anyone's likely to have posted, they're things that must be calculated, every time, for your specific trip.

HERE Location Reasoning gives AI agents tools to compute routes, travel times, reachable areas, and other spatial outcomes using HERE location intelligence.

The two prompt sets and the rules of the run

Before the results, a note on the dataset: we’re publishing it so you can test how your model and harness perform with real location tools. It offers a shared way for anyone building location-aware agents to measure the same questions. To test this, we’ve built two prompt sets:

• A 240-prompt tool capability set testing whether an agent can recognize and solve distinct types of spatial problems, spanning geocoding and search, isoline routing, matrix routing, routing, tour planning and traffic. Each runs at four levels of difficulty

• A consumer prompt set testing the same underlying capabilities through a less structured way people phrase requests

Two rules governed every run. Built-in web search was disabled, as verified in the execution traces and each prompt ran in isolation to prevent the final answer from being truncated.

How we kept from grading ourselves generously

Publishing scores is easy. What makes them worth trusting is the process behind them.

Having a tool available doesn't prove it was used. So we checked two things before we ever asked whether an answer was good. Did the model actually call a location tool and do the places it returned corroborate each other? Scoring was blind, with randomized answer order, a 1–5 scale and a judge from a different model family than the systems being tested. A second pass stripped the source from retrieved data entirely so the judge scored the evidence, not the branding behind it.

We also found a flaw in our rubric. An early rule meant to penalize made-up details flagged real map data such as names, addresses and ratings as fabricated because it seemed too specific to be guessed. The fix was making the rule symmetric to penalize invention in both directions, not just one. It's a small bug, but it only shows up if you're actually trying to break your own scoring, which was the point.

What the scores showed

All scores are mean judge scores on a 1 to 5 scale, from our own runs. On the consumer set, the same model went from 3.01 without location tools to 3.56 with them and to 3.73 once we shipped usage guidance alongside the tools. We started with three frontier models and the results barely moved between them, so the numbers here come from one.

Discovery is where we have the most room to improve. The same model moved from 2.04 to 3.31 once we added location tools which is a bigger lift than any other category but it still trails routing and trip planning, and that gap is real. Part of the reason is structural as no model's training data is current now. Someone asks it a question whether a place is still open or still exists, are exactly the kind that go stale first. We saw this directly. Asked about a specific restaurant, a model confidently described its atmosphere, menu and hours in detail. The restaurant had closed recently:

On the 240-prompt capability pack, where each prompt exercises a specific kind of tool, the overall mean was 4.58 with usage guidance and 4.52 without:

Search along a route, traced call by call

Scores only tell you so much, so here is what one prompt looks like from the inside. This one asks for a lot at once.

Prompt:

That is four jobs in one sentence. The agent has to plan a route for a truck, read the traffic along it, find fuel near the destination and choose one. Each job needs a different tool, and each depends on what the last one returned.

The route came back at 384 km and about 3 hours 29 minutes, using a truck profile of roughly 20 tons gross weight with 1.5 minutes of traffic delay. The traffic reading covered the whole corridor. It sampled 137 road segments with an average jam factor of 0.6 on a scale of 0 to 10. Flagged two minor incidents, a long running construction zone on I-45 near Corsicana and a short slowdown on the north side of Houston. The agent then searched near Houston, got five stations back and picked the one nearest the city center, 370 meters away.

Traffic is a live reading, so yours will differ. Every number in the answer traces back to a tool call you can open and that is the first thing to check when you run the pack. Did your agent call a routing tool or did it predict a drive time based on the weights of the model?

Run the pack against your own stack

We have published the full 240-prompt test pack. Run it against your own agent, on whatever model and harness you're already using and see what changes once it has real location tools to work with.

We're not trying to score you against us. A harness tuned for a smaller model might do more with less than you'd expect and that's worth seeing. Did you find an interesting difference or a harness trick that closes a gap? We’d like to hear about it, share with us on our developer community on Slack. We're in #here-location-reasoning channel and always up for comparing notes.

This evaluation has a few limits. One model judged another, each prompt ran once and the rubric reflects our judgment of a good answer. Treat the results as directional, not ground truth.

Conclusion

Fluency doesn’t guarantee correctness. For questions about the physical world, you see the difference when you check. That’s what this exercise tested: giving an agent real location tools and seeing what changes.

Tools improved every category, including discovery, the category with the most room still to close. That gap is where we're focused next.

Download the test prompts from the HERE GitHub.