Next: Benchmarking Techniques, Previous: Additional Timing Utilities, Up: Profiling [Contents][Index]
There are two general kinds of timing information provided by the
time macro and other profiling utilities: real time and run
time. Real time is elapsed, wall clock time. It will be affected in
a fairly obvious way by any other activity on the machine. The more
other processes contending for CPU and memory, the more real time will
increase. This means that real time measurements are difficult to
replicate, though this is less true on a dedicated workstation. The
advantage of real time is that it is real. It tells you really how
long the program took to run under the benchmarking conditions. The
problem is that you don’t know exactly what those conditions were.
Run time is the amount of time that the processor supposedly spent running the program, as opposed to waiting for I/O or running other processes. “User run time” and “system run time” are numbers reported by the Unix kernel. They are supposed to be a measure of how much time the processor spent running your “user” program (which will include GC overhead, etc.), and the amount of time that the kernel spent running “on your behalf.”
Ideally, user time should be totally unaffected by benchmarking conditions; in reality user time does depend on other system activity, though in rather non-obvious ways.
System time will clearly depend on benchmarking conditions. In Lisp benchmarking, paging activity increases system run time (but not by as much as it increases real time, since the kernel spends some time waiting for the disk, and this is not run time, kernel or otherwise.)
In my experience, the biggest trap in interpreting kernel/user run time is to look only at user time. In reality, it seems that the sum of kernel and user time is more reproducible. The problem is that as system activity increases, there is a spurious decrease in user run time. In effect, as paging, etc., increases, user time leaks into system time.
So, in practice, the only way to get truly reproducible results is to run with the same competing activity on the system. Try to run on a machine with nobody else logged in, and check with “ps aux” to see if there are any system processes munching large amounts of CPU or memory. If the ratio between real time and the sum of user and system time varies much between runs, then you have a problem.
Next: Benchmarking Techniques, Previous: Additional Timing Utilities, Up: Profiling [Contents][Index]