All writing

The model invented a city

New rules took Sninat's search from 18.8% to 63.8% of test searches fully right. Adding an AI model scored higher but made up cities.

Code on GitHub

A wrong city is worse than none

Sninat is a directory of dentists in Morocco. When someone searches, it first works out 3 things from what they typed: what they’re looking for (a dentist, an article or an answer to a practical question), which city, and which kind of care, like braces or implants. Then it only searches what matches that city and kind of care.

So a city it gets wrong costs more than a city it misses.

read with no city read as Fès Rabat Fès Casablanca Rabat Fès Casablanca dentist found dentist left out
Someone whose dentist is in Rabat searches. Read with no city, the search covers every city and that dentist can come up. Read as Fès, it only covers Fès. The dashed line is what gets searched.

People in Morocco search in French, Arabic, English and Darija, the Arabic spoken in Morocco. Darija is often typed in Latin letters, with digits standing in for sounds Latin letters don’t have, and mixed with French or English in the same search: “tbib snan f casa lyoum” means a dentist in Casablanca today. The rules Sninat had only knew French words, spotted them even inside other words, and asked an AI model to guess what the person wanted whenever they found nothing.

The test set

Measuring this needed searches with known right answers. An AI language model (an LLM) drafted 160 in the ways people write, each with its answers, and I checked every one and fixed the ones that were off. Its Darija needed fixing in places: braces are “slk dsnan”, gums are “ltha” and a wisdom tooth is “drsa d l39al”. A few searches were written to catch habits the old rules were known to have, like finding “tanger” inside “tangerine”, which pulls their score down.

Two of them, with the answers each one expects:

[
  {
    "text": "ltha dyali kadmmi",
    "language": "darija-latin",
    "intent": "mixed",
    "city": null,
    "need": "periodontics"
  },
  {
    "text": "tangerine flavored toothpaste for kids",
    "language": "en",
    "intent": "mixed",
    "city": null,
    "need": null
  }
]

The 160 were split in half once, with the same mix of languages in each. I tuned the fixes on one half. The other half wasn’t used to tune anything and was scored once, at the end, and every number here comes from it.

How the old rules did

The old rules got all 3 right on 18.8% of those searches. Every Arabic search went to the AI model, because the rules had no Arabic, and the model only guessed what the person wanted, never the city. Darija did worst. The old rules got 2 cities wrong, both by finding a city’s name inside a longer word: “fes” in “professionnel” and “tanger” in “tangerine”.

3 ways to fix it

New rules. The same decisions, made on whole words. They treat the different ways of writing the same Arabic letter as one, read Darija’s digits as the sounds they stand for, and know the Arabic, Darija and English names of cities and kinds of care. They never ask the AI model.

The model. One question to the AI model Sninat was already using, asking for all 3 answers at once in a fixed format. An answer that isn’t one of Sninat’s cities or kinds of care is thrown away.

Rules, then model. The new rules go first, and the model is asked only when they find no city.

a search new rules city found answer from rules no AI call no city ask the model model's intent and city kind of care from rules
How rules, then model answers a search. When the new rules find no city, the model's intent and city are used, and the rules' kind of care is kept if they found one. All 3 of its wrong cities came in on that branch.

In the repo these are the baseline, A, B and C.

all 3 right wrong city old rules 18.8% 2 new rules 63.8% 0 the model 58.8% 3 rules, then model 72.5% 3
The share of the 80 searches scored at the end that each version got fully right, with a square for each search it gave the wrong city.

Here’s each answer on its own, and how often each version asked the AI model:

VersionWhat they wantCityKind of careAI calls per 100 searches
Old rules56.3%72.5%55.0%48
New rules70.0%98.8%92.5%0
The model75.0%93.8%78.8%100
Rules, then model77.5%96.3%92.5%50

The model was better than the new rules at working out what people wanted. By language, it did best of all on searches that mix Darija with French or English, and worst on Darija alone.

old rules French 25.0% Arabic 12.5% Darija 6.3% English 28.6% Mixed 22.2% new rules 81.3% 81.3% 56.3% 64.3% 38.9% the model French 68.8% Arabic 56.3% Darija 18.8% English 71.4% Mixed 77.8% rules, then model 87.5% 87.5% 43.8% 85.7% 61.1%
Searches with all 3 answers right in each language, one small chart per version. The model's bars are filled in.

All of the model’s wrong cities came from Darija: 3 searches that named no city came back as Casablanca, Salé and Casablanca. Rules, then model got the same 3 wrong, because a search with no city is when it asks the model. The model also takes time. Half its answers took more than 1.7 seconds, and 1 in 20 took 2.8 seconds or more.

What went in

Before the final scoring I set a bar for rules, then model: more searches fully right than the old rules, and no more wrong cities. It got 58 of 80 fully right against 15, and got a city wrong 3 times against 2, so it didn’t go in.

The new rules got 51 fully right, never got a city wrong and don’t call the AI model, and they’re what Sninat’s search runs now. I picked them after seeing these numbers, so their result flatters them a little more than a blind pick would.

Checking where the city ends up turned up something else. Profiles store the city the way it’s displayed, lowercased, so Fès is saved as “fès”, and the search compared it to “fes” exactly. A search naming Fès, Salé, Kénitra, Meknès, Témara or Béni Mellal found no dentists there. It was fixed in the same change, by putting both sides through one function:

export function normalizeCity(city: string): string {
  return city.normalize('NFD').replace(/[\u0300-\u036f]/g, '').toLowerCase().trim()
}

// what a dentist's profile stores, and what a search filters on
city: dentist.city ? normalizeCity(dentist.city) : undefined,
if (options?.city) filter.city = { $eq: normalizeCity(options.city) }

What the numbers cover

They measure whether the search understood what was typed. Whether the right dentists then show up is a separate question.

It’s also one AI model, the one Sninat was already using: Meta’s Llama 3.1 8B, which is small and cheap to run. A bigger model might not make up cities, and none was tested here.

Each half is 80 searches, so a single search moves a score by more than a point, and the gap between the 2 versions’ wrong cities was one search.

The repo has every search and its answers, the code for each version and the model’s replies, so the scores can be run again without a Cloudflare account.