The help assistant
searched the company's policy pages.
Forty of them.
The team kept it simple.
One page,
one embedding,
one vector in the index.
The pages were short anyway,
someone said.
A few screens each.
The demo went well.
Questions about annual leave
found the leave page.
Questions about expenses
found the expenses page.
Then the real questions came.
Can I carry over unused leave?
How do I claim mileage
for a trip that starts at home?
The assistant found the right page
and answered as if
it said nothing about either.
Both answers were there,
on the right page,
halfway down.
The model was not the problem.
The input never reached it.
The embedding model was
all-MiniLM-L6-v2,
a popular choice,
small and fast.
Its model card says it plainly.
By default,
input longer than 256 word pieces
is truncated.
Word pieces are
words and fragments of words.
256 of them
is roughly a couple of hundred words of English.
Each page went in whole.
Each page came out
as a vector of its opening paragraphs.
Everything below
was cut before the model saw it.
No error.
No warning.
A vector came back every time,
the right length,
looking exactly like a good one.
Some hosted embedding services
refuse an oversized input
with an error.
Many local models
just shorten it
and carry on.
Either way,
the limit belongs to the model,
not to your pipeline,
and your pipeline is the one
that has to respect it.
So find the number.
Read the model's maximum sequence length
from its card or its configuration,
not from memory.
Count tokens
with the model's own tokenizer,
because characters and words
are not what it counts.
Split every document
into chunks that fit,
with a little overlap.
Then measure.
Log how many inputs
arrive over the limit.
The right number is zero,
and you should know
if it ever stops being zero.
And build your test questions
from the bottom of documents,
not the top.
The opening paragraph
is the part that always works.
A retrieval system
is only as good
as what was actually embedded.
Check what went in
before you tune what comes out.
– Serguey Asael Shinder
Top comments (0)