Server virtualization: how to avoid, find and eliminate bottlenecks?

Question

Server virtualization is very important these days, so I have been instructed to install some of our software on a virtualized server and see what happens. In short: rsync transfer quickly causes the virtualized server to kneel. A virtualization host is a muscular machine with no other load; I do not think this should happen. Top shows the average values ​​of the average load, and cpu iowait - about 100%. There is a huge bottleneck somewhere.

I am more a programmer than a system administrator, I lack knowledge on how to fix this outside of random Googling. I suspect I'm not alone in this.

What I would like to see here are general recommendations on virtualization, as well as pointers to good articles and other resources that I and others can use for training.

• What tools (even standard unix tools) can be used to identify bottlenecks?
• What indicators should be observed to ensure smooth operation?
• What things can be effectively virtualized?
• What settings are doomed to failure?

I apologize for the breadth of the question. I simply do not have the knowledge to ask useful specific questions about this.

Edit: More about my specific issue:

• XAN paravirtualization, 3 x guest CentOS
• All guests on local SCSI disks have a fully hardware RAID controller.
• rsyncd runs on 1 guest OS, the transfer is initiated from a remote, non-virtualized server via LAN with a bandwidth of 100 Mbps

As I said, I really can not provide a lot of useful data. I do not expect to get a direct solution to this problem, I would be pleased with pointers to where to start creating the skillset needed to better understand these problems.

Answer 1

I was going to put this in a comment, but I think it is more useful in opening:

Could you add more details about your setup:

• Which VM server?
  (VMware Server, VMware ESX, MS VirtualServer, MS Hyper-V, something else?)
• What OS for guests (guests)?
  (Windows, Linux, 32-bit, 64-bit?)
• Where are guest (s) stored?
  (Local disk or NAS or SAN?)
• Have you been rsyncing between guests on the same VM server or between the guest and the physical server?
• If over a network, how fast is the network?

Performance tuning in any environment is 90% of data collection and 10% analysis. Virtualized environments have more variables than physical environments, but more importantly, they have a different response curve than physical environments. Some applications may work better in a virtualized environment than in a physical environment; others will not. You must understand the requirements of the application, as well as the limitations of the implementation.

I do not think that there are any software applications that cannot be successfully deployed on virtual servers if you pay attention to the details. (Applications that require special hardware that cannot be successfully virtualized are another problem.)

Answer 2

I use rsync to sync some parts of our (very new) virtual environment without any problems. I don't think this is a virtualization problem, as it is an I / O problem that you seem to have already identified.

I found that virtualization is very, very burdened with hard drives, and this is only getting worse, the more guests you have on the host box. For machines with very intense I / O, consider segmenting their disk access from other hosts. Do you use any SAN technology? We found this to be very useful at my workplace (we use two 8-core Sun Intel servers and a 1-TB 12-disk iSCSI array).

Is your hardware a fully supported virtualization software provider? If you are trying to run on unsupported hardware, then there is a good chance that your disk controller will not use the best drivers, which explains the slow access to the disk.

You can use iostat on Linux / Unix to get some feedback from I / O, and there is iotop too, although it is not yet packaged in many distributions.

Answer 3

By hand, I would say this is an I / O problem. In virtualized environments, the biggest factors affecting performance are the state of the host machine disk. What we do to optimize performance:

• Fixed disk allocation. This way you get an adjacent block of disk space in which the VM is located.
• Schedule defragmentation of the VM Slice operating system and host server. Fragmentation is your enemy.
• Make sure you end server sessions gracefully. When you jump from a virtual OS, DO NOT just turn off the VM slice, as this causes huge disk fragmentation. The virtual OS needs to complete the stop / restart process.