This is by no means a complete solution, but it may help in conjunction with other measures: There is an I/O scheduler much like a process scheduler, and it can be tweaked.
Most notably, you can actually choose amongst different schedulers:
~# cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
~# echo "deadline" > /sys/block/sda/queue/scheduler
~# cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq
~#
deadline
may help you get more strongly reproducible results.
noop
, as its name implies, is insanely dumb, and will enable you to wreck absolute havoc on I/O performance with little effort.
anticipatory
and cfq
both try to be smart about it, though cfq
is generally the smarter of the two. (As I recall, anticipatory
is actually the legacy scheduler from right before the kernel started supporting multiple schedulers.)
man hdparm
take a look at the -X option maybe? There are quite a few things in there you could use to slow down your drive, but some of them risk doing nasty things to the data!