Tracing disk performance visually in linux

speeding_1379651cMonitoring performance in Linux can wind up being a very boring proposition to some.  It typically involves a lot of command line work and interpretation by the person running the commands.  If you’re a command line junkie who’s really into iostat, sar, blktrace and bonnie++ then this stuff might be somewhat sacrilegious to you :).   If not- read on!

 

So typically your basic I/O monitoring consists of watching iostat and maybe vmstat as well as another handful of tools that tell you what’s going on with your I/O subsystem.  It can prove to be somewhat challenging to paint a picture to support your assertions that there are in fact bottlenecks or at the very least contention between reads and writes.  Gosh- it sure would be nice to show it visually and start to pick up on patterns and other things you just can’t see (or can be very tedious to sift through) in regular CLI based utilities.  Well, help has arrived!

 

By using a handful of tools freely available to you, there’s an easy way to get what you’re after.  Follow my example tutorial below and hopefully you can apply it to your own scenarios and do something good with it.  Enjoy!

 
Install Seekwatcher, blktrace and other tools needed:

# yum install blktrace python python-dev python-matplotlib cython gcc mencoder png2theora
# cd /var/tmp
# wget http://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz
# tar xvzf tip.tar.gz
# cd seekwatcher-b392aeaf693b/
# python setup.py install

 
Generate a report of disk performance:

# seekwatcher -t dd.trace -o output.png -p 'dd if=/dev/zero of=/images/bigfile bs=1M count=4096' -d /dev/sdc1

 

This command will write 4 gigabytes of zero’s to the file /images/bigfile.  Then it will monitor the disk /dev/sdc1 which is where that filesystem resides.  The output of the command will create a file called output.png that looks like this (click on the image for a bigger size):

find
 
 
Generate a video of disk performance:

# seekwatcher -t dd.trace -o dd.mpg -p 'sync && echo 3 > /proc/sys/vm/drop_caches ; dd if=/dev/zero of=/images/bigfile bs=1M seek=500 count=4096' -d /dev/sdc1 --movie

 

This command is also writing 4 gigabytes of zero’s to the same file, however this time I told it to seek 500mb into the disk before writing (remember seek=write, skip=read with dd).  I also have a command running in another window that is reading another file at the same time to generate some seeks and read activity so you can see it.  Note that I’m flushing out the buffer cache to disk right before the dd command this time to avoid any potential of the buffer cache keeping physical writes from occurring. It could be useless as I don’t know for sure if the dd command is interacting with the filesystem layer and therefore the buffer cache when called in this manner. I just figured it would be a safe way to ensure the writes were physical and not virtual so they would show up here. The output of this command will create a file called dd.mpg that looks like this (my apologies, I can’t seem to figure out how to embed a video from my google drive so you’ll have to click on the link):

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s