Rails 4.1: ActiveRecord::Base.find_each Enumerator

enumerator enumerable activerecord edge rails blocks

I was working with several maintenance tasks that query external webservices for a collection of ActiveRecord objects. In order to avoid hitting the webservices' rate limit, we pause every other iteration for a fraction of a second before we continue.

The code looks something like this:

# old_code.rb
class ExternalService

  def self.work_with_users
    rate_limiter = 0
    User.find_each do |user|
      # Do stuff with each user
      rate_limiter += 1
      sleep 0.8 if rate_limiter % 10 == 0
    end
  end

end

I didn't like that the information of our rate limit guard clauses was so scattered: there were bits before the block and others in the block. I had the urge to refactor it into a more concise form using a Ruby block itself.

My idea was to create a method that takes the rate limit, the sleep time and an enumeration and then yield the elements of the enum to a block.

I wrapped that method into a module and out came this:

# rate_limiter.rb
module RateLimiter
  extend ActiveSupport::Concern

  def rate_limit *args, &block
    self.class.rate_limit(*args, &block)
  end
  
  module ClassMethods
    def rate_limit allowed_iterations, sleep_time, enum
      enum.each_with_index do |data, runs|
        yield data
        sleep sleep_time if (runs+1) % allowed_iterations == 0
      end
    end
  end
end

This is how the specs look like:

# rate_limiter_spec.rb
require 'spec_helper'

describe RateLimiter do
  let(:enum) { 1..10 }
  subject { klass.new }

  describe '.rate_limit' do
    it 'accepts three parameters' do
      expect(klass.method(:rate_limit).arity).to eql 3
    end

    it 'yields to a block' do
      expect do |var|
        klass.rate_limit 1, 0.0, enum, &var
      end.to yield_successive_args(*enum.to_a)
    end

    it 'sleeps every n iterations' do
      limit = 3

      klass.stub(:sleep)
      klass.should_receive(:sleep).exactly(enum.size / limit).times

      klass.rate_limit limit, 10.0, enum, &Proc.new{}
    end
  end

  describe '#rate_limit' do
    it 'forwards do its singleton version' do
      klass.should_receive(:rate_limit).with 10, 1.0, enum
      subject.rate_limit 10, 1.0, enum, &Proc.new{}
    end

    it 'yields to a block' do
      expect do |var|
        subject.rate_limit 1, 0.0, enum, &var
      end.to yield_successive_args(*enum.to_a)
    end
  end

  def klass
    @klass ||= Class.new { include RateLimiter }
  end
end

Unfortunately and unlike other enumerable methods, the batch finder ActiveRecord::Base.find_each does not return an Enumerator when called without a block. Not in Rails 3.2 and not in Rails 4.0.

Luckily, it has been solved in Edge Rails (see 840c552) and I quickly created a Rails 4.1.0.beta app to go for the following code:

# external_service.rb
class ExternalService
  include RateLimiter

  def self.work_with_users
    rate_limit 20, 2.0, User.find_each do |user|
      # Do stuff with each user.
      # Pause 2 seconds every 20 iterations
    end
  end
end

ActiveRecord::Base.find_each now returns a proper Enumerator that can be passed into the rate limiter. If you want to learn more about Enumerators in Ruby (or if you'd like to refresh your memory), I can recommend this excellent Ruby Tapas episode by Avdi Grimm.

You can also find all the code examples on Github.

Rails 4.1: ActiveRecord::Base.find_each Enumerator Making use of a new feature in Edge Rails to build a simple rate limiter

Rails 4.1: ActiveRecord::Base.find_each Enumerator
Making use of a new feature in Edge Rails to build a simple rate limiter