We have a VMWare 5.0 environment with 7 Dell PowerEdge R710s with dual 6 core CPUs and 192GB each. For storage we have a NetApp FAS3210 SAN and currently using 1GB iSCSI. We have dual SAN controllers with 2 1GB uplinks for 4GB total uplink, and each of the 7 hosts have 4 1GB ports dedicated to SAN traffic. The SAN traffic is on it's own VLAN and we are currently sharing the same Cisco 4510 switches for SAN traffic and network traffic even though keep in mind they are VLAN'd. Every port is setup for jumbo frames.
Anyway, we have around 130 VMs running in this environment, 90% of them are Windows 2008 with the remaining 10% being Ubuntu machines. We have this new Ubuntu 126.96.36.199 LTS machine we are using for what we call a data warehouse using Postgresql. This machine is having extremely poor performance disk wise. When it started basically our programmers when they would have queries running that were writing data, reads basically froze, in fact if they were writing a bunch of data you could even hardly do an LS command from the shell it would just freeze. After we had somebody come out to look over everything, they had us instead of using a VMDK for the data drive they had me create a raw mapping to a LUN on the SAN and that helped quite a bit, fixed the freezing anyway. However things are still way under where they need to be, we are getting about 100 TPS in Postgresql when running a DD test. We talked to somebody else yesterday who suggested that instead of using a single 2TB LUN for the data, that we make 4 smaller LUNs and then use LVM in Ubuntu to make 1 big logical drive, to give it more disk queues. We've been experimenting with that today and got it to 250ish TPS but still not where we need to be. We did a test and we are comparing to an old data warehouse we have on a standalone server on our old 4GB fiber EMC SAN, Warehouse 4 is the new VM and Warehouse is just the old one. The DD Test for example on the new VM gave 220/304 TPS and 71/28 TPS with 10 connections and 10 transactions. Compared to the old standalone server on the old SAN we got 319/581 and 375/636. We did more test and got similar results, some even much worse than that.
So we are basically getting up to 1/16th the performance on the new VM compared to the old standalone server. So my question is, what are we doing wrong, overlooking, what do we need to do to get this thing working correctly? Programming thinks the problem is we went with iSCSI and we are now looking and I guess going ahead and moving to 10GB iSCSI but I'm thinking there must still be something in the OS level we are overlooking. ANy help appreciated, thanks!