Monitoring performance in Linux can wind up being a very boring proposition to some. It typically involves a lot of command line work and interpretation by the person running the commands. If you’re a command line junkie who’s really into iostat, sar, blktrace and bonnie++ then this stuff might be somewhat sacrilegious to you :). If not- read on!
So typically your basic I/O monitoring consists of watching iostat and maybe vmstat as well as another handful of tools that tell you what’s going on with your I/O subsystem. It can prove to be somewhat challenging to paint a picture to support your assertions that there are in fact bottlenecks or at the very least contention between reads and writes. Gosh- it sure would be nice to show it visually and start to pick up on patterns and other things you just can’t see (or can be very tedious to sift through) in regular CLI based utilities. Well, help has arrived!
By using a handful of tools freely available to you, there’s an easy way to get what you’re after. Follow my example tutorial below and hopefully you can apply it to your own scenarios and do something good with it. Enjoy!
Install Seekwatcher, blktrace and other tools needed:
# yum install blktrace python python-dev python-matplotlib cython gcc mencoder png2theora # cd /var/tmp # wget http://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz # tar xvzf tip.tar.gz # cd seekwatcher-b392aeaf693b/ # python setup.py install
Generate a report of disk performance:
# seekwatcher -t dd.trace -o output.png -p 'dd if=/dev/zero of=/images/bigfile bs=1M count=4096' -d /dev/sdc1
# seekwatcher -t dd.trace -o dd.mpg -p 'sync && echo 3 > /proc/sys/vm/drop_caches ; dd if=/dev/zero of=/images/bigfile bs=1M seek=500 count=4096' -d /dev/sdc1 --movie
This command is also writing 4 gigabytes of zero’s to the same file, however this time I told it to seek 500mb into the disk before writing (remember seek=write, skip=read with dd). I also have a command running in another window that is reading another file at the same time to generate some seeks and read activity so you can see it. Note that I’m flushing out the buffer cache to disk right before the dd command this time to avoid any potential of the buffer cache keeping physical writes from occurring. It could be useless as I don’t know for sure if the dd command is interacting with the filesystem layer and therefore the buffer cache when called in this manner. I just figured it would be a safe way to ensure the writes were physical and not virtual so they would show up here. The output of this command will create a file called dd.mpg that looks like this (my apologies, I can’t seem to figure out how to embed a video from my google drive so you’ll have to click on the link):