Hard filters are quietly deleting your best search results
A product I work on ranks people against a brief written in plain language. Someone types what they are looking for and the system returns a shortlist with reasons attached.
The demo version is easy. Production broke in ways I did not predict, and the single biggest improvement came from deleting code.
The filter that looked like a feature
We extracted entities from the brief: job title, skill, industry. Then we used them as hard filters on the vector store. That is the obvious design and it is wrong.
A hard filter on a free text job title excludes everyone who words it differently. Ask for a platform engineer and you lose the infrastructure engineers, the site reliability people, and the person whose title is three words of internal jargon.
The filter does not fail loudly. It returns fewer, cleaner looking results. The ones it removed are invisible, and they are disproportionately the interesting ones, because unusual people have unusual titles.
The fix: those are not filters, they are query terms. They go into the query text and ranking decides how much each matters. Hard filters are reserved for facts with exactly one correct answer: location, date, whether a record is complete.
That single change improved results more than anything I did to the model.
The order that actually matters
If you are building retrieval over real documents, this is my ranking of what moved the needle, highest first:
Chunk by structure, not by length. A document split every eight hundred characters cuts a role in half and glues the end of one job to the start of the next. Split on the document's own sections and the retrieved passage is a unit a person would recognise.
Cheap filters before expensive ones. Narrow with plain database conditions first, then rank what survives with vector search. Semantic search across everything and filtering afterwards is slower and worse.
Verify every claim against its source. Anything the model asserts gets checked against the text it came from, and anything untraceable is dropped before a human sees it. A score with no provenance is an opinion with a number attached.
Meter cost per tenant from day one. You cannot tune what you do not measure. I capped the expensive per candidate scoring to the top twenty of a result set after watching what a three hundred record search actually costs. Nobody noticed the cap. Everybody would have noticed the bill.
The choice of embedding model was the smallest of these by a wide margin, which is the opposite of where most of the discussion goes.
The trick: harvest the taxonomy from the error messages
One data provider we query has a large set of valid filter values and publishes no list of them. Guessing is miserable, and a model guessing is worse, because it will invent a plausible parameter and you will not find out until the results are quietly wrong.
Their error responses, though, are specific. Send something invalid and the API tells you what it would have accepted.
So I harvested the taxonomy from the errors. A loop that probes, reads the rejection, records the accepted values, and builds the table the documentation does not contain. An afternoon, and it replaced an open ended guessing problem with a lookup.
Two things that make this work in practice: probe with values close to what you think is right, because near misses produce the most informative rejections, and store the result with a date, because it is a snapshot of a system that will change.
When you are working against an undocumented system, its error messages are the most honest documentation it has. They are the only part that cannot be out of date.