Sunspot-Solr slows down to the beast when my application rises to> 1000 objects [Solr Logs Enabled]

I am wondering if anyone noticed any problems with scaling with Sunspot-Solr. Even if I remove all the parameters available for search, and its just counting against the raw class by itself; It still takes 5 to 8 seconds to load my local, 4 to 5 seconds during production.

Has anyone else been able to scale Sunspot-Solr? What are some common problems?

How can one look deeper inside?

Here are the Solr logs for a single request:

Solr Select (208.1ms) {:rows=>20, :start=>0, :q=>"*:*", :sort=>"score desc", :fq=>["type:Organization", "published_b:true", "updated_at_d:[2009\\-02\\-03T16\\:11\\:55Z TO *]"]} Solr Select (5.5ms) {:rows=>20, :start=>0, :q=>"*:*", :sort=>"score desc", :fq=>["type:Organization", "published_b:true", "updated_at_d:[2009\\-02\\-03T16\\:11\\:55Z TO *]"]} Solr Update (12.6ms) <?xml version="1.0" encoding="UTF-8"?><add><doc><field name="type">User</field><field name="type">ActiveRecord::Base</field><field name="id">User 2</field><field name="class_name">User</field><field name="first_name_s">Bob</field><field name="created_at_d">2009-09-28T21:00:27Z</field><field name="last_name_s">Marley</field><field name="email_s"> bob.marley@gmail.com </field><field name="name_s">Bob Marley</field><field name="last_name_text">Marley</field><field name="first_name_text">Bob</field><field name="email_text"> bob.marley@gmail.com </field><field name="name_text">Bob Marley</field></doc></add> Solr Update (487.7ms) <?xml version="1.0" encoding="UTF-8"?><commit/> Completed in 12632ms (View: 11633, DB: 228) | 200 OK [http://localhost/organizations/search] 
+4
source share
2 answers

1000 objects is a children's game for Solr, so there is something fishy where ~ 200 ms of Solr is read. However, your most immediate problem is that you write Solr during what appears to be a GET request - what does this have to do with it? Do you save a searchable object that runs the Sunspot auto index? If you need to update models during a GET request (which should probably be done in a background job, if possible), you need to turn off automatic indexing in Sunspot:

 searchable :auto_index => false # sunspot setup end 

Then you need to explicitly call my_model.index in your controllers when you really want to update them in Solr.

Finally, this big update at the end is the Solr compilation, which tells Solr to write the undefined changes to disk and load a new crawler that reflects these changes. Charges are expensive; Sunspot :: Rails defaults to committing at the end of any request that it writes to Solr, but this behavior is aimed at the principle of least surprise for new Sunspot users, and not for a real production application. You want to disable it in your config/sunspot.yml :

 auto_commit_after_request: false 

Then you probably want to configure autoCommit in your solr/conf/solrconfig.xml - it is commented out in the default Sunspot Solr distribution, and there is an explanation. I found that once a minute is a good place to start.

After making these changes, I will see if your readings are still slow - I think it is quite possible that the reason for this is that every time you perform a search, your write / commit to Solr necessitates loading with a new crawler from disk . Thus, it cannot allow any internal cache to heat up, etc., and, as a rule, is under tremendous stress.

Hope this helps!

+19
source

When I experienced lengthy requests during the update, I came across this blog

mytechmembank.blogspot.de

and it turned out that I had to change the following:

  Performance killer: <str name="buildOnCommit">true</str> Way to go: <str name="buildOnCommit">false</str> 

in solrconfig.xml file

+1
source

Source: https://habr.com/ru/post/1338262/


All Articles