Boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

Question

I am trying to get django to upload static files to S3, but istead I get a 403 forbidden error and I'm not sure why.

Full stacktrace:

Traceback (most recent call last): File "manage.py", line 14, in <module> execute_manager(settings) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv self.execute(*args, **options.__dict__) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute output = self.handle(*args, **options) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 351, in handle return self.handle_noargs(**options) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 89, in handle_noargs self.copy_file(path, prefixed_path, storage, **options) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 184, in copy_file if not self.delete_file(path, prefixed_path, source_storage, **options): File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 115, in delete_file if self.storage.exists(prefixed_path): File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 209, in exists return k.exists() File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 391, in exists return bool(self.bucket.lookup(self.name)) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 143, in lookup return self.get_key(key_name, headers=headers) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 208, in get_key response.status, response.reason, '') boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden 

Content settings.py:

 import os DIRNAME = os.path.dirname(__file__) # Django settings for DoneBox project. DEBUG = True TEMPLATE_DEBUG = DEBUG ADMINS = ( # ('Your Name', '[e-mail]'), ) MANAGERS = ADMINS DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'. 'NAME': os.path.join(DIRNAME, "box.sqlite"), # Or path to database file if using sqlite3. 'USER': '', # Not used with sqlite3. 'PASSWORD': '', # Not used with sqlite3. 'HOST': '', # Set to empty string for localhost. Not used with sqlite3. 'PORT': '', # Set to empty string for default. Not used with sqlite3. } } # Local time zone for this installation. Choices can be found here: # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name # although not all choices may be available on all operating systems. # On Unix systems, a value of None will cause Django to use the same # timezone as the operating system. # If running in a Windows environment this must be set to the same as your # system time zone. TIME_ZONE = 'America/Denver' # Language code for this installation. All choices can be found here: # http://www.i18nguy.com/unicode/language-identifiers.html LANGUAGE_CODE = 'en-us' SITE_ID = 1 # If you set this to False, Django will make some optimizations so as not # to load the internationalization machinery. USE_I18N = True # If you set this to False, Django will not format dates, numbers and # calendars according to the current locale USE_L10N = True # Absolute filesystem path to the directory that will hold user-uploaded files. # Example: "/home/media/media.lawrence.com/media/" MEDIA_ROOT = '' # URL that handles the media served from MEDIA_ROOT. Make sure to use a # trailing slash. # Examples: "http://media.lawrence.com/media/", "http://example.com/media/" MEDIA_URL = "d1eyn4cjl5vzx0.cloudfront.net" # Absolute path to the directory static files should be collected to. # Don't put anything in this directory yourself; store your static files # in apps' "static/" subdirectories and in STATICFILES_DIRS. # Example: "/home/media/media.lawrence.com/static/" STATIC_ROOT = os.path.join(DIRNAME, "static") # URL prefix for static files. # Example: "http://media.lawrence.com/static/" STATIC_URL = "d280kzug7l5rug.cloudfront.net" # URL prefix for admin static files -- CSS, JavaScript and images. # Make sure to use a trailing slash. # Examples: "http://foo.com/static/admin/", "/static/admin/". ADMIN_MEDIA_PREFIX = '/static/admin/' # Additional locations of static files STATICFILES_DIRS = ( # Put strings here, like "/home/html/static" or "C:/www/django/static". # Always use forward slashes, even on Windows. # Don't forget to use absolute paths, not relative paths. os.path.join(DIRNAME, "main", "static"), ) # List of finder classes that know how to find static files in # various locations. STATICFILES_FINDERS = ( 'django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'django.contrib.staticfiles.finders.DefaultStorageFinder', ) # Make this unique, and don't share it with anybody. SECRET_KEY = '<snip>' # List of callables that know how to import templates from various sources. TEMPLATE_LOADERS = ( 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader', 'django.template.loaders.eggs.Loader', ) MIDDLEWARE_CLASSES = ( 'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', ) ROOT_URLCONF = 'DoneBox.urls' TEMPLATE_DIRS = ( # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". # Always use forward slashes, even on Windows. # Don't forget to use absolute paths, not relative paths. os.path.join(DIRNAME, "main", "templates"), os.path.join(DIRNAME, "templates"), os.path.join(DIRNAME, "basic", "blog", "templates"), ) INSTALLED_APPS = ( 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.sitemaps', # Uncomment the next line to enable the admin: 'django.contrib.admin', # Uncomment the next line to enable admin documentation: 'storages', 'django.contrib.admindocs', 'main', 'contacts', 'piston', 'registration', # 'contact_form', 'basic', 'basic.blog', ) # A sample logging configuration. The only tangible logging # performed by this configuration is to send an email to # the site admins on every HTTP 500 error. # See http://docs.djangoproject.com/en/dev/topics/logging for # more details on how to customize your logging configuration. LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler' } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'DEBUG', 'propagate': True, }, 'django.db.backends': { 'handlers': ['mail_admins'], 'level': 'DEBUG', 'propagate': True, } } } DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' AWS_ACCESS_KEY_ID = '<snip>' AWS_SECRET_ACCESS_KEY = '<snip>' STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage' AWS_STORAGE_BUCKET_NAME = "donebox-static" STATIC_FILES_BUCKET = "donebox-static" MEDIA_FILES_BUCKET = "donebox-media" ACCOUNT_ACTIVATION_DAYS = 7 EMAIL_HOST = "email-smtp.us-east-1.amazonaws.com" EMAIL_HOST_USER = '<snip>' EMAIL_HOST_PASSWORD = '<snip>' EMAIL_PORT = 587 EMAIL_USE_TLS = True TEMPLATE_CONTEXT_PROCESSORS = ( "django.contrib.auth.context_processors.auth", "django.core.context_processors.debug", "django.core.context_processors.i18n", "django.core.context_processors.media", "django.core.context_processors.static", "django.contrib.messages.context_processors.messages", "DoneBox.main.context_processors_PandC", ) 

Content requirements .pip:

 django==1.3 django-storages==1.1.4 django-registration==0.8 django-piston==0.2.3 django-tagging==0.3.1 django-extensions==0.8 BeautifulSoup==3.2.1 boto==2.4.1 mysql-python==1.2.3 tweepy==1.9 feedparser==5.1.2 pycrypto==2.6 

Searching for this exception in google does not cause anything interesting. I suspect that I misconfigured the situation, although I'm not sure. Can someone point me in the right direction? Thank you for your time and attention.

Answer 1

I use Amazon IAM for a specific key identifier and access key and just ran into the same forbidden 403 ... Turns out you need to grant permissions that target both the root of the bucket and its subobjects:

 { "Statement": [ { "Principal": { "AWS": "*" }, "Effect": "Allow", "Action": "s3:*", "Resource": ["arn:aws:s3:::bucket-name/*", "arn:aws:s3:::bucket-name"] } ] } 

Answer 2

I would recommend that you try to verify your AWS credentials separately to check if the credentials really have permission to read and write data to the S3 bucket. The following should work:

 >>> import boto >>> s3 = boto.connect_s3('<access_key>', '<secret_key>') >>> bucket = s3.lookup('donebox-static') >>> key = bucket.new_key('testkey') >>> key.set_contents_from_string('This is a test') >>> key.exists() >>> key.delete() 

You should try the same test with a different bucket ('donebox-media'). If this works, the permissions are correct and the problem is with the code or configuration of the Django repository. If it is not with 403, then either:

• Access_key / secret_key lines are invalid
• access_key / secret_key is correct, but this account does not have the required permissions to write to the bucket

Hope this helps. Please report your results.

Answer 3

I had the same problem and finally I found that the real problem was SERVER time. This was misconfigured, and AWS responded with 403 FORBIDDEN.

With Debian, you can autoconfigure with NTP:

ntpdate 0.pool.ntp.org

Answer 4

This will also happen if your computer settings are incorrect.

Answer 5

It is also possible that the wrong credentials are being used. To check:

 import boto s3 = boto.connect_s3('<your access key>', '<your secret key>') bucket = s3.get_bucket('<your bucket>') # does this work? s3 = boto.connect_s3() s3.aws_access_key_id # is the same key being used by default? 

If not, take a look at ~/.boto , ~/.aws/config and ~/.aws/credentials .

Answer 6

In case this helps someone, I had to add the following configuration entry for collectstatic to work and not return 403:

 AWS_DEFAULT_ACL = '' 

Answer 7

Another solution that avoids the use of custom policies and the use of predefined AWS policies:

• Add S3 permissions for your S3 user.

  • IAM / Users / Permissions and Attach Policy
  • Add AmazonS3FullAccess Policy

Answer 8

Here is a clarification with minimal permissions. In all cases, as discussed elsewhere in s3:ListAllMyBuckets , it is necessary on all buckets.

In the default configuration, django storages will upload files to S3 with shared permissions - see Amazon S3 backend django storages

The trial and error showed that in this configuration, by default, only two permissions are needed: s3:PutObject to load the file first and s3:PutObjectAcl to set permissions for this object publicly.

No additional action is required, because from this point, in any case, the reading will be publicly available.

IAM user policy - public (default):

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::bucketname/*" } ] } 

It is not always advisable to have public objects. This is achieved by setting the appropriate property in the settings file.

Django.py settings:

 ... AWS_DEFAULT_ACL = "private" ... 

And then s3:PutObjectAcl no longer required, and the minimum permissions are as follows:

IAM User Policy - Private:

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject" ], "Resource": "arn:aws:s3:::bucketname/*" } ] } 

Answer 9

You may not have access to the bucket you are trying to find / get / create.

Remember: the bucket names must be unique for the entire S3 ecosystem , so if you try to access (lookup / get / create) a bucket named "test", you will not have access to it.