Equivalent to find_each in DataMapper

Question

Is there a way to make the equivalent of ActiveRecord#find_each in DataMapper ?

( find_each will find_each over the query result, loading things in memory in batches of 1000, rather than loading everything into memory)

Answer 1

I checked dm-chunked_query as @MichaelKohl suggested, but I couldn’t get it to work as I expected, it will get the whole collection (I expect it to use OFFSET + LIMIT). So I wrote my own extension, it is quite simple, hope this helps:

 class DataMapper::Collection def batch(n) Enumerator.new do |y| offset = 0 loop do records = slice(offset, n) break if records.empty? records.each { |record| y.yield(record) } offset += records.size end end end end # Example Model.all(:order => :id.asc).batch(1000).each { |obj| p obj } 

Answer 2

I don’t have too many DMs, but it would not be so difficult to write your own, assuming that the DM allows you to manually apply your own “constraints” and “offsets” to queries.

Check out the implementation of find_each / find_in_batches in AR, just a couple of dozen lines.

https://github.com/rails/rails/blob/master/activerecord/lib/active_record/relation/batches.rb#L19

https://github.com/rails/rails/blob/master/activerecord/lib/active_record/relation/batches.rb#L48