I’ve been wandering around the internet for hours looking for something to show which process is doing all the IO on my system, and that will run on my centos 5.3.  It has a kernel which just missed out on per process IO accounting (in /proc/PID/io) and so, most tools (iotop, etc) will not work.

Something like this:

iostat -x -m -d sdk 1

will show how much io is being used on the device (in this case sdk), but not per process.

This, in CentOS, will show some nice IO stats, but again not per process:

dstat

In later releases of CentOS, this will apparently show per process IO:

dstat -s --top-io --top-bio

I finally came accross this blog, and it worked!  However, there is a bug in the perl script which a user has reported via a comment.  I confirmed this, for my system at least, but the comments on the blog seem to have broken.  Here is a copy of the script which works for DIRTY counts:

#!/usr/bin/env perl
# This program is part of Aspersa (http://code.google.com/p/aspersa/)

=pod

=head1 NAME

iodump - Compute per-PID I/O stats for Linux when iotop/pidstat/iopp are not available.

=head1 SYNOPSIS

Prepare the system:

  dmesg -c
  /etc/init.d/klogd stop
  echo 1 > /proc/sys/vm/block_dump

Start the reporting:

  while true; do sleep 1; dmesg -c; done | perl iodump
  CTRL-C

Stop the system from dumping these messages:

  echo 0 > /proc/sys/vm/block_dump
  /etc/init.d/klogd start

=head1 AUTHOR

Baron Schwartz

=cut

use strict;
use warnings FATAL => 'all';
use English qw(-no_match_vars);
use sigtrap qw(handler finish untrapped normal-signals);

my %tasks;

my $oktorun = 1;
my $line;
while ( $oktorun && (defined ($line = <>)) ) {
   my ( $task, $pid, $activity, $where, $device );
   ( $task, $pid, $activity, $where, $device )
      = $line =~ m/(\S+)\((\d+)\): (READ|WRITE) block (\d+) on (\S+)/;
   if ( !$task ) {
      ( $task, $pid, $activity, $where, $device )
         = $line =~ m/(\S+)\((\d+)\): (dirtied) inode (\d+) \(.*?\) on (\S+)/;
   }
   if ( $task ) {
      my $s = $tasks{$pid} ||= { pid => $pid, task => $task };
      ++$s->{lc $activity};
      ++$s->{activity};
      ++$s->{devices}->{$device};
   }
}

printf("%-15s %10s %10s %10s %10s %10s %s\n",
   qw(TASK PID TOTAL READ WRITE DIRTY DEVICES));
foreach my $task (
   reverse sort { $a->{activity} <=> $b->{activity} } values %tasks
) {
   printf("%-15s %10d %10d %10d %10d %10d %s\n",
      $task->{task}, $task->{pid},
      ($task->{'activity'}  || 0),
      ($task->{'read'}      || 0),
      ($task->{'write'}     || 0),
      ($task->{'dirtied'}     || 0),
      join(', ', keys %{$task->{devices}}));
}

sub finish {
   my ( $signal ) = @_;
   if ( $oktorun ) {
      print STDERR "# Caught SIG$signal.\n";
      $oktorun = 0;
   }
   else {
      print STDERR "# Exiting on SIG$signal.\n";
      exit(1);
   }
}

Save it as ‘iodump’.  The header of the script tells you how to run it.  You have to turn on kernel messages about IO:

echo 1 > /proc/sys/vm/block_dump

and then you can run it like this:

while true; do sleep 1; dmesg -c; done | perl iodump

Don’t forget to turn off the IO logs once you’ve finished:

echo 0 > /proc/sys/vm/block_dump

I like to run it in watch a separate window (e.g. in screen):

watch -tn 1 'i=0; while (( ++i < 5 )); do sleep 1; dmesg -c; done | perl iodump'

This will keep an updated view of the output, updating every 5 seconds.