After upgrading to a gigabit (GbE) switch and recycling my old 32-bit D-Link DGE-500T, I found my NFS performance to be absolutely abysmal. I was pulling around 1.9 MB/s over NFS from my file server. That’s poor, even for 100Mbps Ethernet, and surely for 1000Mbps.
After lots of testing, I discovered that using NFS with UDP on my network kills performance. Since my network is only used by myself, I find it especially strange that NFS over UDP works so poorly. I wonder if my cheap Netgear GS605 Gigabit switch isn’t to blame. Using a crossover cable to my file server, NFS works fine with UDP or TCP. The source machine is running Linux 2.6.4, and the target is running Linux 2.6.7. NFS version is 3.
In either case, I finally settled on using NFS over TCP with 16KB rsize and wsize, in /etc/fstab.
192.168.0.2:/home/shared /home/shared nfs async,tcp,vers=3,rsize=16384,wsize=16384,bg,intr,hard 0 3
Now I can pull around 10.5MB/s, which is what I had expected.
Not having another GbE card to test with, I used `nc` to copy files across the network from the file server to multiple boxes with 100Mbps interfaces. Strangely, I only received around 8MB/s to three different hosts simultaneously. The RAID 5 disk array on the file server, attached to a 64-bit 33MHz 3Ware 7410, a four drive array, can read 34MB/s. The array should not have been fully saturated.
I’ll have to get another GbE card and see what the maximum transfer speed is I can achieve. I ought to be able to read ~ 30MB/s to a single host, if I toss the data into /dev/null. I guess I’ll have to wait and see.
I haven’t tested the write performance yet, over the network or on the file server itself. Since it’s an older 3Ware card, the 7410, it doesn’t have the higher RAID 5 performance the 7506 series boasts. I imagine the write performance should still be adequate, but I wish I had a nice 7506.
I need to hunt down a nice, 64-bit GbE Intel card on Ebay and perhaps a newer SCSI adapter to replace my aging AMI MegaRAID. I’d like to run software RAID 1 for my OS partitions so I can recover. If the MegaRAID dies, I’m out of luck. Hardware RAID isn’t strictly necessary for running RAID 1 in my configuration.