Improving Perforce Proxy throughput
A lot of my job involves working with codebases stored on far away Perforce servers. Despite us having a 100Mb internet connection, our file sync speeds have historically never exceeded 500K/sec. The other day we finally figured out the solution, so I thought I'd post it up here in case it helps other people.
In a nutshell, the solution involves setting the following 'tunables' on both the proxy and each client:
How it works
What is TCP Window Size?
The first variable 'net.tcpsize' adjusts the TCP receive window size. The TCP window size is part of the TCP protocol. It determines the amount of data that the server will send to the client before it waits for the client to reply with an acknowledgement. If this window size is too small, it can affect the bandwidth over a high-latency connection. Imagine that the latency is 1s, and the window size is 64K. The server will send the first 64K, and then wait a whole second for the client's reply until it will send the next 64K. Hence the transfer rate will be a maximum of 64K/sec. Following this logic, you can work out the maximum possible bandwidth for a given latency and window size using the following formula:
TCP-Window-Size-in-bits / Latency-in-seconds = Bits-per-second-throughput
It turns out that our TCP window size was fixed at 64K, which given our 130ms latency was giving us the maximum 500K/s throughput we were seeing.
for a more detailed explanation see: http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/
How the window size is determined
From Windows Vista onwards, the TCP window size is automatically adjusted by
the OS. Windows XP had a registry key which would limit the maximum window
size, but this is no longer used. There IS a bug which can cause TCP
auto-tuning to not work correctly in some circumstances. The hotfix for this
bug is here: http://support.microsoft.com/kb/983528
Installing this hotfix was actually one of the first things we tried.
There is one way that applications can disable auto-tuning and fix the
window size to a specific value. This is by setting the SO_RCVBUF option on a
socket using setsockopt (this behaviour is documented in
Perforce actually takes the net.tcpsize value and sets that as the SO_RCVBUF size on all sockets. This explains why Perforce's sockets all have a fixed window size of 64K. The fix is to either set net.tcpsize to a higher value, or you can set it to 0, which will allow Windows to set the window size itself using its own auto-tuning algorithm.
Changing the client buffer size
The second variable 'filesys.bufsize' sets the buffer size for read/write operations. We increase this on the client to get around an issue with the implementation of the Perforce Proxy. As the proxy receives data from the remote server, it writes it to disk and simultaneously forwards it to the client which is requesting the file. If the client doesn't receive each piece of data in a timely manner, it will cause the proxy to stall. On Mac and Linux, this doesn't seem to be a problem, but on Windows these stalls do occur, and the effect is a reduction in download speed. Increasing the filesys.bufsize variable to 2M on the client appears to fix these stalls.
Applying the fixes
Setting tunables on the proxy
Details on the different ways to set Perforce tunables are detailed in the following document: http://www.perforce.com/sites/default/files/shields-tunables-paper.pdf
For a full list of tunables see here: http://www.perforce.com/perforce/doc.current/manuals/cmdref/p4_configure.html
Setting the tunables on the proxy is fairly straightforward. Simply execute the following commandline on the proxy:
P4 set –S “Perforce Proxy” P4POPTIONS=”-v net.tcpsize=2M -v filesys.bufsize=2M”
This will store the P4POPTIONS variable in the registry. The P4POPTIONS variable's contents will be added to the p4p.exe commandline when the Perforce Proxy service is started. Here we are setting both net.tcpsize and filesys.bufsize to 2M. Setting the filesys.bufsize isn't actually necessary, but we'll do it anyway for completeness.
Setting tunables on the client
There is only one (permanent) way to set tunables on the client, which is to use a P4CONFIG file. The process for setting up a P4CONFIG file is as follows:
- Create an environment variable called P4CONFIG and set its value to p4config.txt
- Create a text file called p4config.txt with the following contents:
Put the p4config.txt file in the root of your perforce workspace, and the settings will automatically apply to any P4 command you run in that workspace. If you also want the settings to apply in P4V (the Perforce visual client), you need to put a p4config.txt file in the "c:\Program File (x86)\Perforce" directory (next to p4v.exe)
Some additional information on how P4CONFIG files work can be found here: http://www.perforce.com/perforce/doc.current/manuals/cmdref/env.P4CONFIG.html
With these fixes in place, we've seen a dramatic improvement in transfer speeds. We've now got individual syncs running at around 5MB/sec. When looking at the network-usage graph, instead of seeing a flat line, we can now see sawtooth-patterns in the transfer speeds, which is an indication of the TCP protocol reaching the maximum bandwidth and then backing off.
I hope this information has been of some use to you. In an ideal world, Perforce would change the tunable parameter defaults so that we would all get great transfer speeds right out of the box. Maybe this is something that can happen in the future