Chef throws 403 banned intermittently

Question

I use a completely new chef server (not an enterprise), and a working chef client throws a 403 forbidden error approximately every 1 out of 5 starts. Does anyone know why this might happen?

• chef-server 11.0.12 via getchef.com/chef/install
• chef-client 11.12.4 via rubygem
• ruby 2.1.2
• Debian Wheezy 7.5

Error:

 198.xx.xxx.xxx ================================================================================ 198.xx.xxx.xxx Error executing action `create` on resource 'template[/var/www/mysite.com/shared/config/database.yml]' 198.xx.xxx.xxx ================================================================================ 198.xx.xxx.xxx 198.xx.xxx.xxx Net::HTTPServerException 198.xx.xxx.xxx ------------------------ 198.xx.xxx.xxx 198.xx.xxx.xxx 403 "Forbidden" 198.xx.xxx.xxx 198.xx.xxx.xxx 198.xx.xxx.xxx Resource Declaration: 198.xx.xxx.xxx 198.xx.xxx.xxx --------------------- 198.xx.xxx.xxx # In /var/chef/cache/cookbooks/rails/recipes/production.rb 198.xx.xxx.xxx 198.xx.xxx.xxx 40: template node[:rails][:app_root]+"/shared/config/database.yml" do 198.xx.xxx.xxx 41: owner "root" 198.xx.xxx.xxx 42: group "xx-dev" 198.xx.xxx.xxx 43: mode 0775 198.xx.xxx.xxx 44: end 198.xx.xxx.xxx 45: 198.xx.xxx.xxx 198.xx.xxx.xxx Compiled Resource: 198.xx.xxx.xxx 198.xx.xxx.xxx ------------------ 198.xx.xxx.xxx 198.xx.xxx.xxx # Declared in /var/chef/cache/cookbooks/rails/recipes/production.rb:40:in `from_file' 198.xx.xxx.xxx 198.xx.xxx.xxx template("/var/www/mysite.com/shared/config/database.yml") do 198.xx.xxx.xxx provider Chef::Provider::Template 198.xx.xxx.xxx action "create" 198.xx.xxx.xxx retries 0 198.xx.xxx.xxx retry_delay 2 198.xx.xxx.xxx path "/var/www/mysite.com/shared/config/database.yml" 198.xx.xxx.xxx backup 5 198.xx.xxx.xxx source "database.yml.erb" 198.xx.xxx.xxx cookbook_name "rails" 198.xx.xxx.xxx recipe_name "production" 198.xx.xxx.xxx mode 509 198.xx.xxx.xxx owner "root" 198.xx.xxx.xxx group "xx-dev" 198.xx.xxx.xxx end 198.xx.xxx.xxx 198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] ERROR: Running exception handlers 198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json 198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] ERROR: Exception handlers complete 198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out 198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Net::HTTPServerException: template[/var/www/mysite.com/shared/config/database.yml] (rails::production line 40) had an error: Net::HTTPServerException: 403 "Forbidden" 

Answer 1

Cookbook objects ( template s, cookbook_file s) are loaded by default.

If this is a very long start, and it looks like it is, then the execution time may exceed the maximum time set by the server from the initial authentication.

You can disable lazy loading on the client by adding

no_lazy_load true

to your client .rb. See http: //docs.opscodecode/config_rb_client.html

Answer 2

In addition to Julian's answer, this worked for me when we ran recipes that took more than 15 minutes. 15 minutes - default TTL value for Chef server:

Edit the app.config file under /var/opt/chef-server/erchef/etc or /opt/opscode/embedded/service/opscode-erchef/etc depending on the version of your chef that you installed.

Set s3_url_ttl to something over 900 seconds. We used 3600 to get around the problem and then rebooted the Chef host server.