The import endpoint
took a list of products as JSON.
The lists could be large,
so the client was allowed
to compress them.
Content-Encoding: gzip.
The service decompressed the body,
parsed it,
and saved the rows.
Somebody had thought about abuse.
There was a limit.
Ten megabytes per request,
checked on the way in,
before anything else ran.
One afternoon
a single request
took the service down.
The container hit its memory limit
and was killed.
It restarted,
took the next retry,
and died again.
The request was well under the limit.
It was a few megabytes of gzip.
Inside was mostly
one repeated character.
Here is the mechanism.
Deflate,
the compression inside gzip,
replaces repeated data
with short references back to earlier data.
On data that repeats perfectly,
its ratio approaches
about a thousand to one.
So ten megabytes on the wire
can be ten gigabytes
once you open it.
The limit was real.
It measured the envelope,
not the letter.
The code then did
the most natural thing in the world.
It decompressed the whole body
into memory,
and only then
looked at what it had.
It is called a decompression bomb,
and it is older than most of the code
it still works against.
Do not trust the sizes
the file reports about itself.
The gzip trailer records the length
only modulo four gigabytes,
and the attacker wrote it anyway.
Archive headers
declare an uncompressed size
that can be anything at all.
Count the bytes yourself.
Decompress as a stream,
keep a running total
of what comes out,
and stop the moment it passes
the limit you actually care about.
That limit is on the decompressed size,
because that is what reaches your memory.
Then reject the request
with a clear error.
For archives,
also cap the number of entries,
and refuse an archive
that contains another archive
unless you have a reason to open it.
Every limit you set
should be measured
at the point where the cost lands.
Bytes on the wire
are cheap for both of you.
Bytes in memory
are only cheap for them.
– Serguey Asael Shinder
Top comments (0)