How to handle periodically changing database data in a Rails application?

Question

EDIT: I completely rewrote this question for clarity. I have no comments and answers before.

I support the 2.x Rails application with a lot of statistics. Some data are real and some are estimated for future years. Every year I need to update estimates using real data and calculate new estimates.

I use BIG yml files and migrations to load data into the application every year. My migrations are full of estimates and data adjustments.

---

Problem

My migrations are filled with non-schematic materials, and I can’t even dream of doing db: migrate: reset without waiting for several hours (if it even works). I would like my migrations to be pleasant and clean - only with the changes associated with the scheme. But how can I update the data every year if not using migrations?

---

Help needed

I would like to hear your comments and answers. I am not looking for a silver bullet, more like best practices and ideas, how people deal with this situation.

Answer 1

It looks like you perform a large operation (downloading data using yml files) once a year, but fewer operations once a month.

From my experience with statistics, you are likely to end up doing more and more of these operations to clear and add more data.

I would use a work processing framework like resque and resque scheduler .

You can schedule tasks to run once a month, year, day, or all the time. Work is something like loading yml files (or sets of yml files) or clearing data. You can control the parameters for sending to the task so that you can use one class, but alternate how it updates or cleans your data, depending on how the task is placed or scheduled.

Answer 2

First of all, I have to say that this is a very interesting question. As far as I know, it is not a good idea to download data from migrations. Generally speaking, you should use db / seeds.rb to load data into your db, and I think it would be nice to write a little class helper to add lib to your directory and then call it from db / seeds.rb. I want you to be able to organize your files as follows:

lib/data_loader.rb lib/years/2009.rb lib/years/2010.rb 

Obviously, you should clear your migrations and write the code for lib / data_loader.rb the way you should prefer, but I was only trying to offer a general idea of ​​how I would arrange my code if I have to deal with a problem like this.

I'm not sure that I answered your question in a way that helps, but I hope this happens.

Answer 3

If I were you, I would go for the creation of a custom rake task. You will have access to all your activerecord models and connections, and once a year you will eventually complete:

 rake calculate 

Answer 4

I have a situation where I need to download data from CSV files, which rarely change and daily data is updated from the Internet. I will give a somewhat complete example of how to make the first.

---

First I have a rake file in lib/tasks/update.rake :

 require 'update/from_csv_files.rb' namespace :update do task :csvfiles => :environment do Dir.glob('db/static_data/*.csv') do |file| Update::FromCsvFiles.load(file) end end end 

=> :environment means that we will have access to the database using regular models.

Then I have the code in the lib/update/from_csv_files.rb to do the actual work:

 require 'csv' module Update module FromCsvFiles def FromCsvFiles.load(file) csv = CSV.open(file, 'r') csv.each do |row| id = row[0] s = Statistic.find_by_id(id) if (s.nil?) s = Statistic.new s.id= id end s.survey_area = row[1] s.nr_of_space_men = row[2] s.save end end end end 

Then I can just run rake update:csvfiles whenever the CSV files change to load new data. I also have another task that is configured similarly to updating my daily data.

---

In your case, you should be able to write code to load YML files or do your calculations directly. To handle your smaller fixes, you can create a general method for loading YML files and invoke it using specific files from the rake task. Therefore, you need to include the YML file and update the rake file with the new task. To process the execution order, you can make a rake command that calls other rake tasks in the appropriate order. Now I'm just throwing some ideas, you know better than me.