Benchmarking Ruby


Benchmarking ruby code is essential for improving the performance of applications. The Ruby Standard Library provides a Benchmark module that can be used to measure the running time of any ruby code block.

First, require the Benchmark library:

require 'benchmark'
# => true

Benchmark a single block of code

For benchmarking simple blocks of code, use the #measure method:

puts Benchmark.measure { 10_000_000.times { Object.new } }
#        user     system      total        real
# => 1.280000   0.000000   1.280000 (  1.283235)

The result is in seconds, so instatiating 10 million objects takes a little more than a second. Quite costly!

If we need to print a custom message, we can capture the result and process it:

result = Benchmark.realtime { 10_000_000.times { Object.new } };
puts "Creating ten million objects: #{realtime}s"
# => Creating ten million objects: 1.2911927450004441s

Compare several blocks

We often need to compare several solutions to a certain problem in order to find out which one is best. Let’s say we want to compare the following two functions for finding the n-th fibonacci number:

# DP version
def fib_dp(n)
  (2..n).reduce([0, 1]) { |mem| mem << mem.last(2).reduce(:+) }[n]
end

# Recursive version
def fib_rec(n)
  return 0 if n == 0
  return 1 if n == 1

  fib_rec(n - 1) + fib_rec(n - 2)
end

The unoptimized recursive version is very slow and will choke on values larger than 40. Lets use the #bm method to measure just how much slower it is:

Benchmark.bm(10) do |x|
  x.report('dp:')         { fib_dp(35) }
  x.report('recursive:')  { fib_rec(35) }
end

#                  user     system      total        real
# dp:          0.000000   0.000000   0.000000 (  0.000035)
# recursive:   1.680000   0.000000   1.680000 (  1.671631)

The first argument to #bm defines the label width and larger values will shift the results further right. Anyway, we got what we wanted - the recursive function takes more than a second and a half and the DP version takes a negligible amount of time. In fact, it can calculate very large fibonacci numbers in a very little time:

puts Benchmark.measure { fib_dp(100_000)  }
#        user     system      total        real
# => 0.350000   0.000000   0.350000 (  0.350249)

There is another method in the Benchmark library called #bmbm which runs the tests twice - the first time to warm up the runtime environment and the second time to measure the results. You will want to use this method if you worry that the order of execution of the different code blocks will have an effect on their runtime.

Using benchmark-ips

The benchmark-ips gem provides even more features than the default Benchmark module. Install it and then require it in your program:

require 'benchmark/ips'

Let’s use it to test our previous functions:

Benchmark.ips do |x|
  x.report("dp: ")        { fib_dp(35) }
  x.report("recursive: ") { fib_rec(35) }

  x.compare!
end

# Calculating -------------------------------------
#                 dp:      5.600k i/100ms
#          recursive:      1.000  i/100ms
# -------------------------------------------------
#                 dp:      60.299k (± 2.0%) i/s -    302.400k
#          recursive:       0.517  (± 0.0%) i/s -      3.000  in   5.800686s
#
# Comparison:
#                 dp: :    60299.5 i/s
#          recursive: :        0.5 i/s - 116590.15x slower

The only difference is the x.compare! call at the end, but we get a lot more valuable information about the performance of the two functions - iterations per second, standard deviation and finally a comparison which shows that the unoptimized recursive version is more than 100 000 times slower than the DP version!

There are a few more options available, like setting the time for the warmup and calculation phases or creating a custom suite.

Conclusion and further reading

If you ever wondered if Enumerable#each was faster than a for loop or if [].map.flatten is slower than [].flat_map, take a look at the fast-ruby repo. It contains answers to those questions and to many others.

The Benchmark module and benchmark-ips gem are really easy to use and provide great information about the performance of your code. They are an essential tool in every developers toolbelt.

Related Posts

The Safe Navigation Operator (&.) in Ruby

Ruby 2.3 introduces the &. operator which will make dealing with nils easier

The Null Object pattern

Rails migration generator - how to specify decimal precision and scale

Hi, World

Every techie's first post