All writing

The queue dropped 14 entries and told nobody

Filling a search index stored 90 of 105 entries. Nothing failed loudly, and the count you check lags behind.

Code on GitHub

Sninat is a directory of dentists in Morocco. Its search doesn’t look for the words you typed. It looks for meaning: type “mes gencives saignent” and it should hand you a gum specialist, even though nothing in that dentist’s profile says “saignent”.

That works by turning every dentist, treatment, article and answer into a long list of numbers that stand for its meaning, and keeping those lists in a separate store called an index. A search turns your words into numbers the same way and looks for the closest ones. So the index isn’t a copy of the database. It’s a second thing, built separately, that can drift from the first without anyone noticing.

Filling it for the first time is one call. Here’s what came back:

{"queued":{"dentist":16,"service":67,"faq":8,"article":6,"specialty":8}}

105 entries queued, 104 expected in the index, since one of those articles is a draft and drafts aren’t indexed. A few minutes later the index held 90.

Nothing had failed. The call returned a success, the logs were quiet, and search worked: I typed “orthodontiste casablanca” and got orthodontists in Casablanca. The site looked finished.

Where 14 entries went

a row in the database rows written by SQL stop here a queue message turned into numbers fails 4 times: gone, no record the search index found by a search its own count lags minutes behind
What has to happen before a row can be found by a search. Rows written straight to the database never start the journey, and a message that fails 4 times ends it without leaving a record.

Each entry takes a trip: the app posts a message onto a queue, something picks it up, asks a model to turn the text into numbers, and writes the result to the index. A message that throws is handled the way most queue code handles it: ask for a retry. The queue gives each message 3 retries. After the 4th failure it drops it. A queue can hand its failures to a second queue that keeps them, and this one had no second queue, so there was nowhere for a dropped message to land.

So 14 entries hit a bad minute, probably a burst rate limit on the model, though nothing was recorded, so I can’t tell you. Then they ceased to exist. The database still had all 105 rows. The search index had 90. Nothing in between raised a hand.

Running the same call again fixed most of it, and again fixed the rest:

RunStoredShort by
first9014
second1022
third1040

The count that lags

The obvious check is to ask the index how many entries it holds. That number is behind, sometimes by minutes. Right after the first run it said 8 while searches were already returning dentists that could only have come from the index. Later it sat at 90 for as long as I watched, then jumped.

So a number read straight after a run tells you nothing, in either direction. A number that has stopped moving for a few minutes tells you something.

What actually settled it was counting by kind: 16 dentists, 8 answers, 5 articles, 8 kinds of care. Those are exactly the published rows in the database, which is the comparison that matters. I could only count the small ones that way, since the endpoint caps at 50 results and there are 67 treatments, so the last gap was worked out by subtraction rather than seen.

The failure that doesn’t look like one

Missing entries are the easy problem. They’re absent, and absence is countable.

The other one looks identical to success. Search for “appareil pour redresser les dents” and the site returns results, with confidence, instantly. They were the wrong results, and here’s what that looked like underneath:

Kind of careCloseness
Implantologie0.529
Esthétique Dentaire0.527
Parodontologie0.522
Endodontie0.504

Orthodontics, the actual answer, the one about straightening teeth, wasn’t in the first 4. And look at the spread: the top 4 sit inside 0.025 of each other. When everything is equally close to your question, nothing is close to your question. The ranking is arbitrary, and arbitrary renders exactly like correct.

The cause was dull. Each kind of care is stored with a label, a one-line description, a list of procedure names and a list of reasons to book. It also has a long description, written in sentences: the only text that says orthodontics corrects the alignment of teeth and jaws, which is the thing somebody is describing when they ask for an appareil to redresser their teeth. That field was never included in what got indexed. The index was built from a summary of each page, while people search against what the page says.

Adding it moved the right answer to first:

Kind of careCloseness
Orthodontie0.531
Implantologie0.529
Esthétique Dentaire0.512
Parodontologie0.509

By 0.002. Right, and still not to be trusted, which is the useful part. A model this small, on a free tier, in French, can tell you the order but not with any conviction. The site doesn’t lean on it: plain word rules read the query first, and the meaning search is the fallback for everything they don’t recognise. The fallback got better. It didn’t get good.

What I check now

Three things, because each catches what the others can’t see:

  • Count by kind against the database. Not the index’s own total, which lags, and not “does search work”, which 90 entries answered yes to.
  • Look at the spread, not the top result. If the first and second answers are within a hair, the ranking is noise wearing a confident face.
  • Keep a handful of real searches with the answers written down. I keep 15 of them, in the shape people actually type: “j’ai les dents de travers”, “3endi ltha katdemmi”, “my gums bleed when I brush”. The word rules get 9. The 6 they miss are written down too, because a list of what doesn’t work yet is worth more than a number that says 60%.

The first search I ran after seeding returned dentists in Casablanca and looked perfect. It was built on an index missing 1 entry in 8, ranking by a margin of 0.002, off text that left out the only sentences worth reading. Everything I know about that came from checking. None of it came from the thing telling me.