Repository navigation
SCTP throughput ~8x slower than TCP when the connection has latency #218
Description
Your environment.
- Version: sctp v1.8.2
- Other Information: tested between macOS<–>macOS and macOS<–>debian bullseye
What did you do?
Test setup:
- Throughput tested with @enobufs's sctptest, dependencies updated to use the latest version of pion/sctp
- Latency controlled using macOS "Network Link Conditioner"
- Tested between separate physical devices connected by gigabit ethernet
I ran sctp-test in both TCP and SCTP (sctp-test refers to this as "udp") modes and compared the throughput with varying levels of latency:
- For TCP:
./sctptest -network tcp -s <addr>[client] and./sctptest -network tcp[server] - For SCTP:
./sctptest -network udp -s <addr>[client] and./sctptest -network udp[server]
What did you expect?
I expected the TCP and SCTP throughput to be in the same ballpark.
What happened?
SCTP overwhelmingly underperformed TCP. Approximate performance over gigabit ethernet:
| Latency | TCP | SCTP |
|---|---|---|
| 2ms | 850 Mbps | 850 Mbps |
| 20ms | 800 Mbps | 200 Mbps |
| 50ms | 600 Mbps | 85 Mbps |
| 100ms | 320 Mbps | 45 Mbps |
| 300ms | 105 Mbps | 17 Mbps |
Background
I ran into this performance issue when trying to diagnose why DataChannel performance was very slow when using a TURN server. My TURN server is about 100ms away, and I was seeing very poor performance (< 10 Mbps) relative to the speed of the internet connection.
SCTP.throughput.and.latency.mp4
Reacted by Antoine Baché and yaoxinliu
Activity
iamcalledrob commentedon Mar 26, 2022
I found this issue from a few years back which looks potentially related? #62
Sean-Der commentedon Mar 27, 2022
Fantastic testing @iamcalledrob thanks for reporting!
I don't know the latest but @enobufs mentioned he was looking at adding RACK to the implementation. Maybe some other things exist that we can do in the short term? I defer to him on what the best thing to do is :)
enobufs commentedon Mar 27, 2022
@iamcalledrob Thanks for the detailed report. As I mentioned in #62, a modern TCP implementation automatically adjust receive buffer size, while pion/sctp uses a fixed buffer size, which is currently set to 1024KB. The receive buffer size sets the hard limit for the amount of data inflight (or the un-ack'd segments). It appears the throughput you observed is the result of this limit. I think pion/sctp should implement this auto tuning feature. Let me dig into more information about its algorithm.
iamcalledrob commentedon Mar 28, 2022
@Sean-Der, @enobufs Thank you both for looking into this!
@enobufs regarding the buffer size, I was curious about that as well and decided to do some more testing!
I benchmarked the effect of changing the buffer size via your
sctp-testutility—i.e.sctp-test -network udp -b N. I believe this sets both the read and write buffer sizes to 'N bytes' on the SCTP association as well as the underlying UDP conn.Here's what I saw with round-trip latency of 300ms
As you can see, the throughput maxes out with a buffer size of 256kb—smaller than the default 1mb! Increasing the buffer size beyond 256kb has no effect.
I'm curious about this—I would have assumed that manually setting a larger buffer would have improved throughput in this particular test, but that has not been the case.
I'm happy to help with getting to the bottom of this, though I have to admit I'm not coming from a place of expertise here so pointers are very helpful :)
I've attached trace logs for both the client and server in case it's helpful context.
client.log
server.log
enobufs commentedon Mar 29, 2022
That indicates there would be another limiting factor. From my quick glance, the sender's congestion window kept growing during the 10 sec period but it appears too slow - smell of a bug... I may be wrong. Let me look at this later more carefully.
enobufs commentedon Apr 3, 2022
I just found a bug in sctptest. The value with the
-boption for the server does not propagate down to pion/sctp :( .What's going on is, the client side should have a ssthresh (slow start threshold) which should initially be set to the receive buffer window advertised by the peer. This value did not change even though I was increasing the value with
-b, then found this bug. I will fix it shortly.enobufs commentedon Apr 3, 2022
After fixing the above in sctptest, I was still seeing the throughput was still capped at 1MB receive buffer size. (increasing the size beyond 1MB had no effect).
sctptest RTT=300ms
From the trance log, I noticed that even though there's more room in the congestion window, the inflightQueue was not fully utilized. Then I realized, this was because sctptest has maxBufferedAmount set to 1MB (threshold being 512KB, both hardcoded), which was effectively limiting the send buffer size to 1MB. This is why the throughput did not grow beyond 1MB of receive buffer size.
There was another related issue. The bufferedAmount in the current pion/sctp includes the number of bytes inflight. Looking at the FireFox's implementation, it does not include the size inflight (the size written to usrsctp).
After fixing the all above, now I see the throughput grows proportionally to the size of buffers as shown below:
Other related metrics:
I will write some unit tests then submit a pull-request.
Once all these fixes land, we should consider whether we should dynamically adjust the receive buffer size.
enobufs commentedon Apr 3, 2022
Following updates have been made:
TODO:
iamcalledrob commentedon Apr 19, 2022
Thank you for looking into this! I'm excited to see what the throughput looks like when inflight size is excluded :)
iamcalledrob commentedon Apr 20, 2022
With your updated version of go-rudp/sctp-test (passing buffer size into pion/sctp assoc), I see dramatically better throughput.
For the curious:
This is faster than TCP.
To repro these tests, I had to make a small fix to cap conn.SetReadBuffer/SetWriteBuffer to ~5mb for mac (the OS UDP limit), while allowing the sctp buffer to be the specified size
Very excited for the use-cases this unlocks for pion/sctp!
Sean-Der commentedon Apr 20, 2022
That's so exciting! Nice work @iamcalledrob and @enobufs :)
We should add this to the
SettingEnginetoday and publish the findings!enobufs commentedon Apr 20, 2022
@Sean-Der I think it is a brilliant idea to allow changing the send/recv buffer size from the SettingEngine as I think it may take sometime for us to implement auto buffer size tuning.
Exclude inflight size from bufferedAmount
Max recv buf size via SettingEngine
Added a unit test
Max recv buf size via SettingEngine
Added a unit test
Max recv buf size via SettingEngine
Added a unit test
iamcalledrob commentedon Jun 14, 2022
@enobufs I'm curious if the commit to exclude the in-flight size is ready to land (or could be!)
I've been testing datachannels with the larger buffer offered through the SettingEngine, but am still seeing very low throughout when there's even a tiny amount of packet loss or latency.
Testing with sctptest over LAN and the network link conditioner looked really positive--near 100% link utilisation.
However in the real world I'm still seeing <20% link utilisation over WiFi or coast-to-coast levels of latency.
For example, I have a TURN server which is 80ms away. iperf with a few TCP streams will get 600+Mbps throughput. PION datachannels with an 8mb buffer will get 20Mbps throughput.
I was hoping the larger buffer would be a decent enough fix by itself in the "real world", but that doesn't seem to be the case!
Max recv buf size via SettingEngine
Added a unit test
suconghou commentedon Nov 29, 2022
@enobufs
v1.8.2
poor performance than tcp in a tiny amount of packet loss or latency
does v1.8.3 fixed
pion/sctp: exclude inflight size from bufferedAmount?or improved it .
MousyBusiness commentedon Mar 23, 2023
@Sean-Der Is MaxReceiveBuffer arbitrary or is it supposed to match a UDP socket configuration? What are the tradeoffs if the value is arbitrary?
Max recv buf size via SettingEngine
Added a unit test
Lanius-collaris commentedon Sep 24, 2025
The low utilization problem of RFC 4960 in fast long-distance networks is documented in RFC 8312
Lanius-collaris commentedon Feb 28, 2026
@iamcalledrob @suconghou
Hi, you can try cc-alg branch.
Association.CC1can be replaced.Note that FRCC may take tens seconds to converge.
JoTurk commentedon Feb 28, 2026
We should rerun this, it has been 4 years.
Lanius-collaris commentedon Nov 4, 2026
Test setup:
bandwidth × delayproduct (bandwidth = 120 Mbit/s, minimum RTT = 200 ms, MTU=1500), a not very high loss rate (1/10000), and a infinite queueRun iperf3 with
-C cubicfor 60s.linux-cubic.log:
BBR can utilize more bandwidth, so we transfer 400 Mbytes.
linux-bbr.log:
Use pion/sctp with FRCC to transfer 400 Mbytes too. ( https://github.com/Lanius-collaris/pion-sctp/tree/526574e3da0cac3036b85dc8d0d5f6eb591ea628 )
pion-sctp-frcc.log: