Formerly we sent twice on pollChan, but because it was unbuffered, the
second send was almost always dropped. KCP's own ACK packets should also
serve as another source of what are effectively polling queries at a
rate proportional to the rate at which we are receiving.
I did some performance tests of downloading 10 MiB between two servers
with 100 ms RTT between them. Server:
dnstt-server -udp :53 -privkey-file server.key t.example.com 127.0.0.1:9321
ncat -l -k -v 9321 --send-only --sh-exec 'dd bs=1M count=10 if=/dev/urandom'
Client:
dnstt-client -pubkey-file server.pub t.example.com 127.0.0.1:7000
ncat --recv-only 127.0.0.1 7000 | pv -t -r -a -b -i 0.2 > /dev/null
I did the download under every treatment twice and recorded the download
rate in KiB/s.
First, the results for the commit before this one. I also hacked in
fewer sends on pollChan for each packet received. The result for 1 poll
are about the same as for 2 polls, which is expected, with the
observation that the unbuffered pollChan was usually dropping the second
send. 0 polls results in a very slow rate (possibly driven only by smux
keepalive packets). ("~" means I stopped waiting for the download after
about 2 minutes.)
resolver method pollChan cap SetACKNoDelay polls KiB/s KiB/s
-------- ------ ------------ ------------- ----- ----- -----
direct udp unbuffered false 2 146 159 (status quo before this commit)
dns.google udp unbuffered false 2 58.3 60.2 (status quo before this commit)
dns.google doh unbuffered false 2 125 126 (status quo before this commit)
direct udp unbuffered false 1 155 165
dns.google udp unbuffered false 1 57.5 58.7
dns.google doh unbuffered false 1 124 123
direct udp unbuffered false 0 ~7 ~11
dns.google udp unbuffered false 0 ~6 ~5
dns.google doh unbuffered false 0 ~6 ~6
Now, the result after this commit. A buffered pollChan with 1 poll per
receive has almost identical performance to an unbuffered pollChan with
1 or 2 polls per receive, which is expected. Using 2 polls rather than 1
actually helps performance a fair bit in the direct/UDP and Google/UDP
treatments, but hurts performance in the Google/DoH treatment. 0 polls
still yields poor performance.
resolver method pollChan cap SetACKNoDelay polls KiB/s KiB/s
-------- ------ ------------ ------------- ----- ----- -----
direct udp 16 false 2 174 175
dns.google udp 16 false 2 76.0 75.3
dns.google doh 16 false 2 68.7 68.0
direct udp 16 false 1 149 151 (this commit)
dns.google udp 16 false 1 60.2 60.3 (this commit)
dns.google doh 16 false 1 123 121 (this commit)
direct udp 16 false 0 ~7 ~8
dns.google udp 16 false 0 ~5 ~5
dns.google doh 16 false 0 ~6 ~7
I tried the additional modification of calling conn.SetACKNoDelay(true).
My guess was that this would cause every received data packet to be
ACKed immediately, which should have the same function as a poll.
Unexpectedly for me, SetACKNoDelay(true) actually slows down the
direct/UDP and Google/DoH cases a lot. But strangely, Google/UDP becomes
faster (and 1 poll is even faster than 2 polls in that case). Strangest
of all, SetACKNoDelay(true) makes the 0-poll Google/UDP and Google/DoH
treatments run reasonably fast. My best guess as to why that is the case
is that routing through a Google resolver tends to disorder the packet
sequence, which maybe results in more ACKs, which effectively act as
polls.
resolver method pollChan cap SetACKNoDelay polls KiB/s KiB/s
-------- ------ ------------ ------------- ----- ----- -----
direct udp 16 true 2 87.7 84.2
dns.google udp 16 true 2 82.7 68.9
dns.google doh 16 true 2 59.8 60.8
direct udp 16 true 1 54.1 47.8
dns.google udp 16 true 1 97.5 100
dns.google doh 16 true 1 71.4 70.9
direct udp 16 true 0 ~9 ~8
dns.google udp 16 true 0 48.0 48.6
dns.google doh 16 true 0 59.2 59.2
Not amount of raw payload. This allows for the case where the received
payload is only padding, for example. (That can't happen with the
current downstream encoding scheme, which doesn't allow for padding, so
I believe this change results in equivalent behavior.)
David Fifield