RubyConf 2007: Profiling and Tuning Ruby 1.8
Is Ruby 1.8 Slow?
To benchmark: Collect a set of benchmark times, then normalize them, then compute the geometric mean of the ratios.
Alioth is a popular (if controversial) set of benchmarks. Using gcc as the standard, java is about 3 times slower, python is about 10, and ruby is 19 times slower (slower than Python, PHP, and Perl).
“Hemibel thinking”: (half of a an “order of magintude” [factor of 10]). Look for hemibel improvements or difference. Any greater accuracy doesn’t really help. Hemibel ratios: log10(a/b) * 2. Now we can rethink java as ~1, perl ~2, ruby ~2.6 (in terms of hemibels). Now we can say “Ruby is sort of slow.”
But, of course, reducing the speed of a language to a single number is not really that interesting. Instead, we need to find the variation of the benchmarks. Cue Box and Whisker Plots (they show a lot of data in a small amount of space). Now yarv looks really nice, with a really limited variation over the different benchmarks (python and ruby do well, perl and php do poorly). Each language has big (upper) outliers. Ruby’s worst outlier is “spectralnorm 500″ (also bad for yarv).
Author: Ed Borasky