Fixed Time Quanta (FTQ)
The FTQ benchmark takes a time based approach, rather than a workload based, approach to analyzing deterministic processing behavior of operating systems.
Simply put, rather than doing a fixed unit of work, and measuring how long that work took; FTQ measures the amount of work done in a fixed time quantum. The time quantum is short, typically around one microsecond, and many measurements are taken. The benchmark is carefully designed to ensure that the time interval is closely adhered to, and the code is also careful to ensure that the sample intervals start at well-defined, periodic intervals. In other words, given a time interval T, we ensure that sampling starts at t, t+T, t+2T, ... and so on. This last property is key to producing good statistics for use in post-processing. We correctly manage the problems that occur when the process is interrupted at, e.g., t+2.5T, and restarted at, e.g., t+5T.
What's the difference, you might wonder, between measuring the time to do a fixed piece of work and the amount of work that can be done in a fixed piece of time? As it happens, it's all the difference in the world! Consider how hard it would be, for example, to reason about audio frequency in terms of seconds per cycle. It would difficult at best to compare different noises by their spectra, as we do now. Instead, once we characterise audio sounds by cycles per second, we can easily compare them. It is natural to think of periodic phenomena in time-based terms.
By making time the independent variable, and work the dependent variable, we can look for frequency artifacts -- such as daemons waking up as the select loop times out -- and their impact on system performance. The time based approach allows for more advanced analysis of the results than workload based benchmarks. For example, with results in the format of "units of work per fixed time period" rather than "amount of time per fixed workload", the analysis can examine the frequency domain and point to periodic problems. In practice we can see interference artifacts of up to several Khz.
FTQ In the Industry
FTQ has found usage in a number of computer firms (including IBM) and is going to be used to characterise future computer systems being purchased by a government agency (more when that is released).
FTQ Visualization Tools
The output if FTQ is raw data in two files, containing time in one and corresponding units of work in the other. Any number of scientific analysis packages can be used to visualize and interpret these results. I have put together a frequency domain visualization tool which plots the time plot and the fast fourier transform of a pair of output files. It uses python, numpy, pylab, and scipy, to present interactive plots.
- python (debian package: python)
- numpy (debian package: python-numeric)
- scipy scipy (debian package: python-scipy)
- pylab (debian package: python-matplotlib)