I’ve been messing around a lot with Xen lately and have seen several different articles and forum posts debating the advantages over using file-based disk images, like /mnt/xen/VM01-disk.img, versus giving the VM direct access to a LVM partition. So, I ran a few simple tests on my own to determine what would be best for my machine.

Dom-0 is using four 1.5tb 7200rpm seagate drives in a software Raid-10, /dev/md2. Both Dom0 and DomU have 1gb of ram and are using ext3. Xen is using mainly default settings with the default scheduler.

From DomU with the DomU image being file-based on ext3 Dom-0 fs - DomU has 1gb ram

[root@DomU]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 192.556 seconds, 55.8 MB/s

From Dom0 on the same filesystem where above DomU’s image is stored - Dom0 has 1gb ram

[root@Dom0]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 72.0743 seconds, 149 MB/s

I was really surprised to see there was this much of a difference between Dom0 and DomU disk access. Several tests were run, and the results didn’t vary much at all.

I also ran a bonnie++ benchmark on both Dom0 and DomU.
Dom0:

 Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
razer.justynshul 2G   419  98 144092  21 68102   2   882  97 183977   2 699.3   0
Latency             60846us    1136ms     381ms   53632us   98757us     371ms
Version  1.96       ------Sequential Create------ --------Random Create--------
razer.justynshull.c -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency                87us     447us     486us     694us      34us      41us
1.96,1.96,razer.justynshull.com,1,1304536826,2G,,419,98,144092,21,68102,2,882,97,183977,2,699.3,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,60846us,1136ms,381ms,53632us,98757us,371ms,87us,447us,486us,694us,34us,41us

DomU:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
razer2.justynshu 2G   446  99 94601  12 62259   1  1006  99 130897   1 182.1   0
Latency             64057us    1771ms    2969ms   19222us    1887ms     348ms
Version  1.96       ------Sequential Create------ --------Random Create--------
razer2.justynshull. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             44083us     395us     410us      59us      10us      66us
1.96,1.96,razer2.justynshull.com,1,1304538856,2G,,446,99,94601,12,62259,1,1006,99,130897,1,182.1,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,64057us,1771ms,2969ms,19222us,1887ms,348ms,44083us,395us,410us,59us,10us,66us

Round 2 – LVM-based disk on DomU

Here are the results of the same tests, except this time DomU is using LVM as the root disk instead of a file-based image. It is still on the same exact hardware and drives/raid set-up.

[root@DomU]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 123.34 seconds, 87.1 MB/s

Bonnie++ test:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
CentOS           2G   444  99 78784   6 48588   1   957  97 190338   1 612.1   0
Latency             18285us     433ms     306ms   33926us   48079us     456ms
Version  1.96       ------Sequential Create------ --------Random Create--------
CentOS              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              6908us     386us     402us     433us      23us      33us
1.96,1.96,CentOS,1,1304828315,2G,,444,99,78784,6,48588,1,957,97,190338,1,612.1,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,18285us,433ms,306ms,33926us,48079us,456ms,6908us,386us,402us,433us,23us,33us

I should re-run the bonnie tests with a large test size since these may be inaccurate with only 2g. Going purely off of the dd results, using LVM is the way to go.
The Sequential Output from bonnie++ is higher for the DomU using a file-based disk image, but that doesn’t match up to any of the dd tests on the same DomU. When I have time, I’ll re-run the bonnie tests with larger file sizes to see if the results stay the same or not.