Summary: When testing ZFS read performance with fio, compression settings on the file system may cause you to test cache performance instead of physical disk performance.
Background: Testing was done on a FreeBSD 8.3-STABLE system, with an eleven-disk, non-root zpool:
$ zpool status
scan: scrub repaired 0 in 0h11m with 0 errors on Fri Jun 14 17:20:12 2013
NAME STATE READ WRITE CKSUM
tank01 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
da11 ONLINE 0 0 0
errors: No known data errors
ZFS and zpool versions were 5 and 28, respectively. Recordsize was 128K. Compression was set to “on”.
Data disks were Toshiba model MK1001TRKB; the separate log was an STEC ZeusRAM C018. Drives were connected to a single LSI SAS2008 HBA.
The system had a single 2.13GHz Intel Xeon E5606 processor, and 12GB of RAM; via
/boot/loader.conf, ARC size was limited to 6GB.
$test types of “read” and “randread”, fio was run as follows, five times for each test:
System activity was monitored during each run using a combination of sysctl, iostat, vmstat and top. Fio “IO file” size was measured using “du -h”.
IO files were then written to using the following command:
After writing to the IO files, tests were re-run, again five times per test.
Fio appears to create new IO files in a highly-compressible format. With compression on, files that have not been written to are 512 bytes in size; without compression, they are the full size specified in the fio command line (36GB in this case). Once the files have been written to, on a file system with compression turned on, they grew in size to about 14GB (i.e. the files had a compression ratio slightly better than 2:1).
Performance was substantially better on files that had not been written to as opposed to those that had been written to; run-to-run variation, as measured by the standard deviation, was larger on the written files:
||mean MB/s unwritten file
||stdev MB/s unwritten file
||mean IOPs unwritten file
||stdev IOPs unwritten file
||mean MB/s written file
||stdev MB/s written file
||mean IOPs written file
||stdev IOPs written file
top values suggest that benchmark performance was bounded by the CPU on unwritten files, and zpool disks on the written files.
sysctl counters and iostat indicated that effectively no reads for the unwritten files were served from disk but came instead from (prefetch) cache; written files exercised the disks, but when data was read from cache for the written files, it came predominantly from the ARC.
The randread/written file results, in aggregate, have far greater variation as a proportion of the achieved performance; looking at individual runs an interesting pattern emerges:
||ARC cache hit ratio
Specifically, performance got better with each run, apparently as a result of ARC caching. Initially, cache hits seem to be drawn from the MRU, but by the fourth and fifth tests, the MFU is more heavily used. The ARC caches uncompressed data, but even at 36GB of file data with a 6GB ARC, it is reasonable that some proportion of “random” read data will be served from the ARC. The possible implication that the ARC is successfully able to adapt to fio’s random read workload would be interesting to look at in greater depth.
Conclusion: Although this is a limited set of data, two results can reasonably be drawn from it:
- The combination of cache and compression in ZFS can have impressive performance benefits; and
- ARC and prefetch cache are each relevant in different performance domains.
Acknowledgements: I am indebted to my former employer for allowing me free usage of the system tested in this blog post.