Run large scripts with rails

Question

I made a very large script to feel my initial data in my rails application. I have about 3,000 lines in CSV and 10,000 images.

After 300 downloads I received this message:

/usr/local/lib/ruby/gems/1.9.1/gems/activesupport-3.0.9/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``': Cannot allocate memory - identify -format %wx%h '/tmp/stream20111104-14788-1hsumv7.jpg[0]' (Errno::ENOMEM) 

My script load:

 if (row[28] != nil) hotelalbum = HotelAlbumPhoto.find_or_create_by_title(h.title) hotelalbum.description = "Album photo de l'hotel " + h.title.capitalize hotelalbum.hotel_id = h.id hotelalbum.save files = Dir.glob('IMAGES/' + row[28].gsub(/\\/,'/') + '/*.jpg') i =0 for file in files i += 1 photopath = File.expand_path('../../import', __FILE__) + '/' + file photoname = file.split('/').last if (i==1) hotelalbum.thumbnail = open(photopath) hotelalbum.save end if (i==1) h.thumbnail = open(photopath) end photo = HotelImage.find_or_create_by_image_file_name_and_hotel_album_photo_id(photoname,hotelalbum.id) if (photo.image_file_size == nil || photo.image_file_name != photoname) photo.image = open(photopath) photo.activated = true photo.alt = "Photo de l'hotel " + h.title photo.save else puts photopath + ' already updated' end end end 

When I check my memory with the top command, I see that the ruby ​​process uses more memory for each boot. How can I manage this?

thanks for the help

ps: My server is a virtual machine with 512 MB memory, one solution is to increase this memory, but I hope to find another.

Answer 1

I don’t know where the open function is defined, but I suspiciously don’t see the corresponding closure ...

update Best idea, change photo.image = open(photopath) to photo.image = File.read(photopath)

In accordance with the documents read:

 Opens the file, optionally seeks to the given offset, then returns length bytes (defaulting to the rest of the file). read ensures the file is closed before returning. 

Answer 2

Sounds like an ImageMagick memory leak problem? Perhaps this will help to process the list in large blocks or fragments using in_groups_of and forcing garbage collection using GC.start after each fragment:

 files.in_groups_of(100) {|chunk| # process chunk GC.start }