In a nutshell, the solution involves setting the following 'tunables' on both the proxy and each client:

        net.tcpsize=2M
    filesys.bufsize=2M


How it works


What is TCP Window Size?

The first variable 'net.tcpsize' adjusts the TCP receive window size. The TCP window size is part of the TCP protocol. It determines the amount of data that the server will send to the client before it waits for the client to reply with an acknowledgement. If this window size is too small, it can affect the bandwidth over a high-latency connection. Imagine that the latency is 1s, and the window size is 64K. The server will send the first 64K, and then wait a whole second for the client's reply until it will send the next 64K. Hence the transfer rate will be a maximum of 64K/sec. Following this logic, you can work out the maximum possible bandwidth for a given latency and window size using the following formula:

TCP-Window-Size-in-bits / Latency-in-seconds = Bits-per-second-throughput

It turns out that our TCP window size was fixed at 64K, which given our 130ms latency was giving us the maximum 500K/s throughput we were seeing.

for a more detailed explanation see: http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/


How the window size is determined

From Windows Vista onwards, the TCP window size is automatically adjusted by the OS. Windows XP had a registry key which would limit the maximum window size, but this is no longer used. There IS a bug which can cause TCP auto-tuning to not work correctly in some circumstances. The hotfix for this bug is here: http://support.microsoft.com/kb/983528
Installing this hotfix was actually one of the first things we tried.

There is one way that applications can disable auto-tuning and fix the window size to a specific value. This is by setting the SO_RCVBUF option on a socket using setsockopt (this behaviour is documented in http://msdn.microsoft.com/en-us/library/windows/desktop/cc136103(v=vs.85).aspx).
Perforce actually takes the net.tcpsize value and sets that as the SO_RCVBUF size on all sockets. This explains why Perforce's sockets all have a fixed window size of 64K. The fix is to either set net.tcpsize to a higher value, or you can set it to 0, which will allow Windows to set the window size itself using its own auto-tuning algorithm.


Changing the client buffer size

The second variable 'filesys.bufsize' sets the buffer size for read/write operations. We increase this on the client to get around an issue with the implementation of the Perforce Proxy. As the proxy receives data from the remote server, it writes it to disk and simultaneously forwards it to the client which is requesting the file. If the client doesn't receive each piece of data in a timely manner, it will cause the proxy to stall. On Mac and Linux, this doesn't seem to be a problem, but on Windows these stalls do occur, and the effect is a reduction in download speed. Increasing the filesys.bufsize variable to 2M on the client appears to fix these stalls.



Applying the fixes


Setting tunables on the proxy

Details on the different ways to set Perforce tunables are detailed in the following document: http://www.perforce.com/sites/default/files/shields-tunables-paper.pdf

For a full list of tunables see here: http://www.perforce.com/perforce/doc.current/manuals/cmdref/p4_configure.html

Setting the tunables on the proxy is fairly straightforward. Simply execute the following commandline on the proxy:

P4 set –S “Perforce Proxy” P4POPTIONS=”-v net.tcpsize=2M -v filesys.bufsize=2M”

This will store the P4POPTIONS variable in the registry. The P4POPTIONS variable's contents will be added to the p4p.exe commandline when the Perforce Proxy service is started. Here we are setting both net.tcpsize and filesys.bufsize to 2M. Setting the filesys.bufsize isn't actually necessary, but we'll do it anyway for completeness.


Setting tunables on the client

There is only one (permanent) way to set tunables on the client, which is to use a P4CONFIG file. The process for setting up a P4CONFIG file is as follows:

  • Create an environment variable called P4CONFIG and set its value to p4config.txt
  • Create a text file called p4config.txt with the following contents:
    filesys.bufsize=2M
        net.tcpsize=2M

Put the p4config.txt file in the root of your perforce workspace, and the settings will automatically apply to any P4 command you run in that workspace. If you also want the settings to apply in P4V (the Perforce visual client), you need to put a p4config.txt file in the "c:\Program File (x86)\Perforce" directory (next to p4v.exe)

Some additional information on how P4CONFIG files work can be found here: http://www.perforce.com/perforce/doc.current/manuals/cmdref/env.P4CONFIG.html


Conclusion

With these fixes in place, we've seen a dramatic improvement in transfer speeds. We've now got individual syncs running at around 5MB/sec. When looking at the network-usage graph, instead of seeing a flat line, we can now see sawtooth-patterns in the transfer speeds, which is an indication of the TCP protocol reaching the maximum bandwidth and then backing off.

I hope this information has been of some use to you. In an ideal world, Perforce would change the tunable parameter defaults so that we would all get great transfer speeds right out of the box. Maybe this is something that can happen in the future



Feedback: #janvanvalburg