You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Increase in memory usage when compression is on #203
I've been started using with Centrifugo in the past week.
I'm using the raw Websocket endpoint which uses this library under the hood.
I'm experiencing a situation where per-message-deflate is enabled there is a massive grow in memory up to the point the docker container crashes for using too much memory.
I'm running inside a docker container, with average 150K-200K concurrent users, my average message rate is between 30K-50K messages per sec, with average message size of 600 bytes.
Without the per-message-deflate the is no memory grow at all and performance is awesome, but the data transfer is very high.
@garyburd I understand that having non-standard lib package in core could be a wrong step, but maybe we can consider mechanism to let it be plugged somehow by user's code?
even without that fix [it] behaves without memory grows... I can't explain this.
AFAICT, this package uses "level 3" compression (which is a good choice). In my package level 1-4 are a specialized, and do not use the "generic" code, which has the issue.
In Go 1.7 level 1 (Best speed) has a similar specialized function. I would think that if you use that, you will not experience the issue. That might be a solution you can use, so you do not have to import a specialized package (even if I wouldn't mind to give users the option). Performance for level 1 should be very close to my package.
@klauspost thanks for explaining, just tried what you said - yes, with compression level 1 performance is comparable to your library and has no memory problems in go1.7 (in test case above)
@garyburd what do you think about this? I see two solutions that can help us - make compression level exported variable or allow to plug custom flate implementation. Of course we can also wait for go1.8 but a way to improve compression performance still very important. Do you want us to try creating custom build with advices you gave and fixed array copy bug and see how it behaves in production?
@garyburd I agree with @FZambia, having the option to set the compression would help greatly, for my specific use case I have a fan out of 80K messages per sec for 200K users which cause a lot of traffic out, if I will be able to set the compression level I could find the sweet spot between servers numbers and traffic out.
Usually, servers cost much less than traffic when the traffic is so high so having this option configurable will be awesome
Thanks
@garyburd I agree that level 1 is better for default value at moment because it fixes memory grows on Go1.7 and compression is so costly. But looks like on fanout level in such a big apps as @joshdvir has saving bandwidth saves a lot of money so having compression level configurable makes sense.
We made a custom build with compression level 1 and counters you suggested and put it into production. Counter values are:
--inuse_space shows similar picture just values two order of magnitude less (2.69GB for flateWriterPool.Get().(*flate.Writer) line which is leader).
It's hard to say what can we do with such a big compression overhead...
@garyburd you suggest adding CompressionOptions - I can do pull request with it, but maybe just a global exported variable like DefaultFlateCompressionLevel that we can set on application start or setter method will do the work? We don't need per-connection compression level - and we can eventually do what you suggested if there will be a need later. And no deprecation will be required this way for now.
I think that having one method that changes default compression level still makes sense:
vardefaultCompressionLevelint=1// SetDefaultCompressionLevel sets the flate compression level which will be used by // default to compress messages when compression negotiated. This function must be // called once before application starts.//// Valid levels range from -2 to 9. Level -2 will use Huffman compression only. // Level -1 uses the default compression level. Level 0 does not attempt any// compression. Levels 1 through 9 range from best speed to best compression. func (c*Conn) SetDefaultCompressionLevel(nint) error {
defaultCompressionLevel=n
}
In most situations I suppose users that need custom compression need to set this default value once.
Then if someone needs compression level per connection/message we can add SetCompressionLevel and a [12]flateWriterPool:
// SetCompressionLevel sets the flate compression level for the next message.// Valid levels range from -2 to 9. Level -2 will use Huffman compression only. // Level -1 uses the default compression level. Level 0 does not attempt any// compression. Levels 1 through 9 range from best speed to best compression.// If not set default compression level will be used.func (c*Conn) SetCompressionLevel(nint) error {
}
If it's not called but compression negotiated defaultCompressionLevel will be used. The only caveat I see is that SetDefaultCompressionLevel should be called once before application starts in current implementation but it seems pretty opaque.
Yeah - there is quite a number of tables needed to maintain state, produce huffman tables and buffer output. There isn't much that can be done about it, except for level -2(HuffmanOnly), 0 (No Compression) and 1(Best Speed) in the standard library.
For my own library I have been looking at reducing the memory requirements for encoding options that does not require as many buffers, see klauspost/compress#70 (it still has issues, as can be seen by the crash I logged in the issue)
@garyburd many thanks for your time helping us with this. Looking at all these profiles do you see any way to reduce compression memory usage and allocations?
@klauspost thanks, I'll try to check it out as soon as I have time. I have a gist that allows to use custom flate with Gorilla WebSocket library - need to revisit it and experiment with your stateless compression branch.
Compressor and decompressor instances are very memory intensive, and gorilla uses sync.Pool to reuse them. For best reuse, messages should be processed in a non-blocking manner, with a goroutine opened immediately after decoding.
@jaitaiwan gzip (deflate) compression mostly relies on referencing previous data. Beside the window (that is typically extended, so it doesn't have to copy too much) it also keeps temporary buffers, so it doesn't have to allocate for every block, as well as index of the window (typically the most memory intensive).
For reference here is the approximate "idle" usage of each encoder level. That means the allocs made to create an instance and the memory used after a Write call before the Writer is closed:
A) Use a Stateless Writer. This works as a normal compressor but de-allocates everything between each Write call. This means there is 0 "idle" allocations, but there is a significant compression loss, and each Write will cause temporary allocations.
B) Do your own buffering of the last data written, and use StatelessDeflate to provide the history. This will result in less compression loss, but still allocs on each write
C) Do your own buffering of the last data written, and use ResetDict before each write and flush+stash the encoders afterwards.
Activity
kisielk commentedon Jan 11, 2017
First step would probably be to get a heap and allocation profile of your application using pprof.
joshdvir commentedon Jan 11, 2017
Thanks guys,
I'm waiting for the next version of Centrifugo which includes your latest version and then will profile the application and upload here.
You can close this issue until then, but I hope to have the new version today or tomorrow the latest so it's up to you.
joshdvir commentedon Jan 11, 2017
Here is the pprof dump
This is the heap profile
@kisielk @garyburd I've updated the gist please check now
https://gist.github.com/joshdvir/091229e3d3e4ade8d73b8cffe86c602b
FZambia commentedon Jan 13, 2017
I asked @joshdvir to send be cpu and memory profiles from production node, here is what we have:
CPU:
Most of cpu time spent in WriteMessage:
NextWriter:
compressNoContextTakeover:
And now heap profile:
NextWriter:
compressNoContextTakeover:
y3llowcake commentedon Jan 14, 2017
Possibly related: golang/go#18625
FZambia commentedon Jan 14, 2017
@y3llowcake thanks for pointing on this issue.
I've written test case for Gorilla Websocket:
It creates 1000 connections with compression enabled, each with buffered message channel. Then in a loop we write message into each connection.
Here is how it behaves with
go1.7.4Using Go with commit golang/go@9c3630f
Though It's hard to say at moment will this fix solve original problem in this issue or not.
I also tried the same with
flatefrom https://github.com/klauspost/compress by @klauspost which already contains that array copy fix in master:But actually even without that fix https://github.com/klauspost/compress library behaves without memory grows... I can't explain this.
Also here is benchmark result using https://github.com/klauspost/compress library:
It's 4x speedup comparing to standard lib
compress/flateresults:@garyburd I understand that having non-standard lib package in core could be a wrong step, but maybe we can consider mechanism to let it be plugged somehow by user's code?
klauspost commentedon Jan 14, 2017
AFAICT, this package uses "level 3" compression (which is a good choice). In my package level 1-4 are a specialized, and do not use the "generic" code, which has the issue.
In Go 1.7 level 1 (Best speed) has a similar specialized function. I would think that if you use that, you will not experience the issue. That might be a solution you can use, so you do not have to import a specialized package (even if I wouldn't mind to give users the option). Performance for level 1 should be very close to my package.
FZambia commentedon Jan 14, 2017
@klauspost thanks for explaining, just tried what you said - yes, with compression level 1 performance is comparable to your library and has no memory problems in go1.7 (in test case above)
FZambia commentedon Jan 17, 2017
@garyburd what do you think about this? I see two solutions that can help us - make compression level exported variable or allow to plug custom flate implementation. Of course we can also wait for go1.8 but a way to improve compression performance still very important. Do you want us to try creating custom build with advices you gave and fixed array copy bug and see how it behaves in production?
joshdvir commentedon Jan 17, 2017
@garyburd I agree with @FZambia, having the option to set the compression would help greatly, for my specific use case I have a fan out of 80K messages per sec for 200K users which cause a lot of traffic out, if I will be able to set the compression level I could find the sweet spot between servers numbers and traffic out.
Usually, servers cost much less than traffic when the traffic is so high so having this option configurable will be awesome
Thanks
FZambia commentedon Jan 17, 2017
@garyburd I agree that level 1 is better for default value at moment because it fixes memory grows on Go1.7 and compression is so costly. But looks like on fanout level in such a big apps as @joshdvir has saving bandwidth saves a lot of money so having compression level configurable makes sense.
We made a custom build with compression level 1 and counters you suggested and put it into production. Counter values are:
It's aggregation over 1 minute. So pool looks pretty effective but...
...Compression is still the leader in allocs and CPU profiles and getting from sync.Pool is the most allocation-expensive operation for some reason.
Here is CPU profile now:
list WriteMessage
list Close
list Flush
Memory usage is much better now but it still very high, compression allocates a lot:
list NextWriter
list compressNoContextTakeover
--inuse_spaceshows similar picture just values two order of magnitude less (2.69GB for flateWriterPool.Get().(*flate.Writer) line which is leader).It's hard to say what can we do with such a big compression overhead...
@garyburd you suggest adding
CompressionOptions- I can do pull request with it, but maybe just a global exported variable likeDefaultFlateCompressionLevelthat we can set on application start or setter method will do the work? We don't need per-connection compression level - and we can eventually do what you suggested if there will be a need later. And no deprecation will be required this way for now.FZambia commentedon Jan 18, 2017
I think that having one method that changes default compression level still makes sense:
In most situations I suppose users that need custom compression need to set this default value once.
Then if someone needs compression level per connection/message we can add
SetCompressionLeveland a [12]flateWriterPool:If it's not called but compression negotiated
defaultCompressionLevelwill be used. The only caveat I see is thatSetDefaultCompressionLevelshould be called once before application starts in current implementation but it seems pretty opaque.FZambia commentedon Jan 18, 2017
Just looked at size of each
flate.Writerinstance:600Kb! This size is surprising for me - I have not even assumed that it's such a big thing:)
klauspost commentedon Jan 18, 2017
Yeah - there is quite a number of tables needed to maintain state, produce huffman tables and buffer output. There isn't much that can be done about it, except for level -2(HuffmanOnly), 0 (No Compression) and 1(Best Speed) in the standard library.
For my own library I have been looking at reducing the memory requirements for encoding options that does not require as many buffers, see klauspost/compress#70 (it still has issues, as can be seen by the crash I logged in the issue)
FZambia commentedon Jan 19, 2017
@garyburd many thanks for your time helping us with this. Looking at all these profiles do you see any way to reduce compression memory usage and allocations?
22 remaining items
nhooyr commentedon Nov 9, 2019
@klauspost that's really cool. What sort of memory overhead would be involved for each Write?
klauspost commentedon Nov 9, 2019
It will require a few hundred kilobytes, but contrary to before the other alternatives once it is compressed the memory will be freed.
For web sockets, that is what you need. Meaning no live allocations for inactive sockets. Just be sure all is written in a single write.
FZambia commentedon Nov 10, 2019
@klauspost thanks, I'll try to check it out as soon as I have time. I have a gist that allows to use custom flate with Gorilla WebSocket library - need to revisit it and experiment with your stateless compression branch.
klauspost commentedon Nov 22, 2019
klauspost/compress#185 brings down heap allocation to a few hundred bytes during a Write by having an internal
sync.Pool.nhooyr commentedon Feb 16, 2020
My testing of @klauspost's library with stateless compression has shown great results.
See klauspost/compress#216 (comment)
About 9.5 KB allocated per message written for the simple 512 byte message
[]byte(strings.Repeat("1234", 128)).edit: It's now available for testing in master on my library.
nhooyr commentedon Feb 16, 2020
To clarify, that's with a dictionary in context takeover mode, it's 50 B a message allocated if there is no dictionary.
lxzan commentedon Sep 28, 2023
Compressor and decompressor instances are very memory intensive, and gorilla uses sync.Pool to reuse them. For best reuse, messages should be processed in a non-blocking manner, with a goroutine opened immediately after decoding.
jaitaiwan commentedon Sep 2, 2024
It's acceptable to add a dependency, but its not clear to me how using this would improve the situation?
klauspost commentedon Sep 4, 2024
@jaitaiwan gzip (deflate) compression mostly relies on referencing previous data. Beside the window (that is typically extended, so it doesn't have to copy too much) it also keeps temporary buffers, so it doesn't have to allocate for every block, as well as index of the window (typically the most memory intensive).
For reference here is the approximate "idle" usage of each encoder level. That means the allocs made to create an instance and the memory used after a Write call before the Writer is closed:
So there are 3 variations:
A) Use a Stateless Writer. This works as a normal compressor but de-allocates everything between each Write call. This means there is 0 "idle" allocations, but there is a significant compression loss, and each Write will cause temporary allocations.
B) Do your own buffering of the last data written, and use StatelessDeflate to provide the history. This will result in less compression loss, but still allocs on each write
C) Do your own buffering of the last data written, and use ResetDict before each write and flush+stash the encoders afterwards.