The model invented a city
New rules took Sninat's search from 18.8% to 63.8% of test searches fully right. Adding an AI model scored higher but made up cities.
A wrong city is worse than none
Sninat is a directory of dentists in Morocco. When someone searches, it first works out 3 things from what they typed: what they’re looking for (a dentist, an article or an answer to a practical question), which city, and which kind of care, like braces or implants. Then it only searches what matches that city and kind of care.
So a city it gets wrong costs more than a city it misses.
People in Morocco search in French, Arabic, English and Darija, the Arabic spoken in Morocco. Darija is often typed in Latin letters, with digits standing in for sounds Latin letters don’t have, and mixed with French or English in the same search: “tbib snan f casa lyoum” means a dentist in Casablanca today. The rules Sninat had only knew French words, spotted them even inside other words, and asked an AI model to guess what the person wanted whenever they found nothing.
The test set
Measuring this needed searches with known right answers. An AI language model (an LLM) drafted 160 in the ways people write, each with its answers, and I checked every one and fixed the ones that were off. Its Darija needed fixing in places: braces are “slk dsnan”, gums are “ltha” and a wisdom tooth is “drsa d l39al”. A few searches were written to catch habits the old rules were known to have, like finding “tanger” inside “tangerine”, which pulls their score down.
Two of them, with the answers each one expects:
[
{
"text": "ltha dyali kadmmi",
"language": "darija-latin",
"intent": "mixed",
"city": null,
"need": "periodontics"
},
{
"text": "tangerine flavored toothpaste for kids",
"language": "en",
"intent": "mixed",
"city": null,
"need": null
}
]
The 160 were split in half once, with the same mix of languages in each. I tuned the fixes on one half. The other half wasn’t used to tune anything and was scored once, at the end, and every number here comes from it.
How the old rules did
The old rules got all 3 right on 18.8% of those searches. Every Arabic search went to the AI model, because the rules had no Arabic, and the model only guessed what the person wanted, never the city. Darija did worst. The old rules got 2 cities wrong, both by finding a city’s name inside a longer word: “fes” in “professionnel” and “tanger” in “tangerine”.
3 ways to fix it
New rules. The same decisions, made on whole words. They treat the different ways of writing the same Arabic letter as one, read Darija’s digits as the sounds they stand for, and know the Arabic, Darija and English names of cities and kinds of care. They never ask the AI model.
The model. One question to the AI model Sninat was already using, asking for all 3 answers at once in a fixed format. An answer that isn’t one of Sninat’s cities or kinds of care is thrown away.
Rules, then model. The new rules go first, and the model is asked only when they find no city.
In the repo these are the baseline, A, B and C.
Here’s each answer on its own, and how often each version asked the AI model:
| Version | What they want | City | Kind of care | AI calls per 100 searches |
|---|---|---|---|---|
| Old rules | 56.3% | 72.5% | 55.0% | 48 |
| New rules | 70.0% | 98.8% | 92.5% | 0 |
| The model | 75.0% | 93.8% | 78.8% | 100 |
| Rules, then model | 77.5% | 96.3% | 92.5% | 50 |
The model was better than the new rules at working out what people wanted. By language, it did best of all on searches that mix Darija with French or English, and worst on Darija alone.
All of the model’s wrong cities came from Darija: 3 searches that named no city came back as Casablanca, Salé and Casablanca. Rules, then model got the same 3 wrong, because a search with no city is when it asks the model. The model also takes time. Half its answers took more than 1.7 seconds, and 1 in 20 took 2.8 seconds or more.
What went in
Before the final scoring I set a bar for rules, then model: more searches fully right than the old rules, and no more wrong cities. It got 58 of 80 fully right against 15, and got a city wrong 3 times against 2, so it didn’t go in.
The new rules got 51 fully right, never got a city wrong and don’t call the AI model, and they’re what Sninat’s search runs now. I picked them after seeing these numbers, so their result flatters them a little more than a blind pick would.
Checking where the city ends up turned up something else. Profiles store the city the way it’s displayed, lowercased, so Fès is saved as “fès”, and the search compared it to “fes” exactly. A search naming Fès, Salé, Kénitra, Meknès, Témara or Béni Mellal found no dentists there. It was fixed in the same change, by putting both sides through one function:
export function normalizeCity(city: string): string {
return city.normalize('NFD').replace(/[\u0300-\u036f]/g, '').toLowerCase().trim()
}
// what a dentist's profile stores, and what a search filters on
city: dentist.city ? normalizeCity(dentist.city) : undefined,
if (options?.city) filter.city = { $eq: normalizeCity(options.city) }
What the numbers cover
They measure whether the search understood what was typed. Whether the right dentists then show up is a separate question.
It’s also one AI model, the one Sninat was already using: Meta’s Llama 3.1 8B, which is small and cheap to run. A bigger model might not make up cities, and none was tested here.
Each half is 80 searches, so a single search moves a score by more than a point, and the gap between the 2 versions’ wrong cities was one search.
The repo has every search and its answers, the code for each version and the model’s replies, so the scores can be run again without a Cloudflare account.