Django-pipe and s3boto repository don't seem to work together

Question

I am trying to use django-pipeline-1.1.27 with s3boto to compress and filter static files and then load them into an s3 bucket. If I just use:

PIPELINE_STORAGE = 'pipeline.storage.PipelineFinderStorage' 

Then it works, and I get a static folder with a nice versioned file that I configured. As soon as i switch to

 PIPELINE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' 

I get

 Traceback (most recent call last): File "manage.py", line 15, in <module> execute_manager(settings) File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv self.execute(*args, **options.__dict__) File "/my/virtual/env/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute output = self.handle(*args, **options) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/management/commands/synccompress.py", line 39, in handle packager.pack_stylesheets(package, sync=sync, force=force) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/packager.py", line 52, in pack_stylesheets **kwargs) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/packager.py", line 60, in pack package['output'], package['paths']) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/__init__.py", line 45, in need_update version = self.version(paths) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/__init__.py", line 20, in version return getattr(self.versioner, 'version')(paths) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 37, in version buf = self.concatenate(paths) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 27, in concatenate return '\n'.join([self.read_file(path) for path in paths]) File "/my/virtual/env/lib/python2.7/site-packages/pipeline/versioning/hash/__init__.py", line 31, in read_file file = storage.open(path, 'rb') File "/my/virtual/env/lib/python2.7/site-packages/django/core/files/storage.py", line 33, in open file = self._open(name, mode) File "/my/virtual/env/lib/python2.7/site-packages/storages/backends/s3boto.py", line 177, in _open raise IOError('File does not exist: %s' % name) IOError: File does not exist: css/style.css 

which is one of my source files. So, why does the pipeline no longer want to execute the filter / concatenate / compress steps when switching to s3boto repository?

Maybe I'm doing something. Here is another configuration if this helps:

 INSTALLED_APPS = ( ... 'pipeline', 'storages', ) STATICFILES_FINDERS = ( 'pipeline.finders.PipelineFinder', 'django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', ) STATIC_ROOT = "/some/path/outside/django_project/deploy_static" STATICFILES_DIRS = () # All statics in this site are in apps STATICFILES_STORAGE = 'pipeline.storage.PipelineStorage' PIPELINE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' PIPELINE = True PIPELINE_AUTO = True PIPELINE_VERSION = True PIPELINE_VERSION_PLACEHOLDER = 'VERSION' PIPELINE_VERSIONING = 'pipeline.versioning.hash.SHA1Versioning' PIPELINE_CSS = { 'standard': { 'source_filenames': ( 'css/style.css', ... ), 'output_filename': 'css/all-VERSION.css', 'extra_context': { 'media': 'screen,projection', }, } } 

My site is on Django 1.3.1.

The command that I run:

 python manage.py synccompress --force 

AWS credit cards are also in the settings, but this is a moot point, because it doesn’t even get to this point.

UPDATE Added full stack and settings requested in comments

UPDATE . At the request of the author of the library, I tried updating to the latest beta. Observations from this so far:

• I do not know how to get compressed files with version now
• collectstatic leaves me with compressed files and originals
• Still getting the same error from the django pipeline when the boto configuration is configured: it wants to send my source files to s3, but I don’t even see where it places my assets. Nothing fits in STATIC_ROOT.

UPDATE I created a simple project that works for the crawler repository and then splits into S3Boto. I pushed it to github and enabled stacktrace capture.

https://github.com/estebistec/simple_pipeline https://raw.github.com/estebistec/simple_pipeline/master/STACKTRACE

I would be delighted if I were told that I am doing some really dumb ones, and that should all work.

Answer 1

django-pipeline 1.1.x little dumb about how you should use staticfiles, it prefers to have everything in one place. I suggest you try django-pipeline 1.2 with the latest django-staticfiles or django 1.4 .

Use a user like this:

 STATICFILES_STORAGE = 'your.app.S3PipelineStorage' 

The code is as follows:

 from staticfiles.storage import CachedFilesMixin from pipeline.storage import PipelineMixin from storages.backends.s3boto import S3BotoStorage class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage): pass 

You can find how to fix your application, but there is still an error with compiled files if you are not using version 1.2c1: https://gist.github.com/1999564

Answer 2

I just experienced the same error in a Django 1.6 project with django-pipeline==1.3.23 , and the solution just PIPELINE_STORAGE parameter.

Answer 3

Another problem with a similar error message that affects the previous and current version (1.5.4) of django-pipeline .

The error message is IOError: File does not exist , and this happens in s3boto.py.open() and packager.pack_stylesheets() . You may encounter a problem if you use any of the compilers (Compass, Sass, Less, etc.). I suspect that this would also affect the JS compiler, but I have not confirmed.

In a nutshell, the compiler creates the output file in the local static storage, and the next steps compress tries to find the output in the s3 storage.

If this affects you, you can take a look at https://github.com/cyberdelia/django-pipeline/issues/473 . There are two pull requests (patches), one of which is made by skirsdeda, and the other by thomasyip (me). Both can solve your problem. If you want the compiled (but pre-compressed) file to be copied to s3 and available for the application, you should take the thomasyip (me) patch.

Here's the full Traceback for the problem:

 Traceback (most recent call last): File "apps/manage.py", line 16, in <module> execute_from_command_line(sys.argv) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line utility.execute() File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv self.execute(*args, **options.__dict__) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute output = self.handle(*args, **options) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/management/base.py", line 533, in handle return self.handle_noargs(**options) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 171, in handle_noargs collected = self.collect() File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 117, in collect for original_path, processed_path, processed in processor: File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/storage.py", line 26, in post_process packager.pack_stylesheets(package) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/packager.py", line 96, in pack_stylesheets variant=package.variant, **kwargs) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/packager.py", line 106, in pack content = compress(paths, **kwargs) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 73, in compress_css css = self.concatenate_and_rewrite(paths, output_filename, variant) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 137, in concatenate_and_rewrite content = self.read_text(path) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 220, in read_text content = self.read_bytes(path) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 214, in read_bytes file = staticfiles_storage.open(path) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/django/core/files/storage.py", line 35, in open return self._open(name, mode) File "/Users/thomas/Dev/Project/main-server/venv/lib/python2.7/site-packages/storages/backends/s3boto.py", line 366, in _open raise IOError('File does not exist: %s' % name) IOError: File does not exist: sheets/sass/sheets.css 

Answer 4

In addition to the answers, you can use GZIP also when compressing:

 from django.contrib.staticfiles.storage import CachedFilesMixin from pipeline.storage import PipelineMixin from storages.backends.s3boto import S3BotoStorage class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage): def __init__(self, *args, **kwargs): self.gzip = True super(S3PipelineStorage, self).__init__(*args, **kwargs) 

Using the following settings:

 COMPRESS_STORAGE = STATICFILES_STORAGE = 'my.apps.main.S3PipelineStorage' 

Answer 5

Not sure how this seemed to work for everyone else. I followed the solution above and kept getting the following error:

 Traceback (most recent call last): File "manage.py", line 24, in <module> execute_from_command_line(sys.argv) File "python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line utility.execute() File "python3.4/site-packages/django/core/management/__init__.py", line 330, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "python3.4/site-packages/django/core/management/base.py", line 390, in run_from_argv self.execute(*args, **cmd_options) File "python3.4/site-packages/django/core/management/base.py", line 441, in execute output = self.handle(*args, **options) File "python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle collected = self.collect() File "python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect for original_path, processed_path, processed in processor: File "python3.4/site-packages/pipeline/storage.py", line 26, in post_process packager.pack_stylesheets(package) File "python3.4/site-packages/pipeline/packager.py", line 96, in pack_stylesheets variant=package.variant, **kwargs) File "python3.4/site-packages/pipeline/packager.py", line 105, in pack paths = self.compile(package.paths, force=True) File "python3.4/site-packages/pipeline/packager.py", line 99, in compile return self.compiler.compile(paths, force=force) File "python3.4/site-packages/pipeline/compilers/__init__.py", line 56, in compile return list(executor.map(_compile, paths)) File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 549, in result_iterator yield future.result() File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 402, in result return self.__get_result() File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result raise self._exception File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run result = self.fn(*self.args, **self.kwargs) File "python3.4/site-packages/pipeline/compilers/__init__.py", line 42, in _compile outdated = compiler.is_outdated(input_path, output_path) File "python3.4/site-packages/pipeline/compilers/__init__.py", line 85, in is_outdated return self.storage.modified_time(infile) > self.storage.modified_time(outfile) File "python3.4/site-packages/storages/backends/s3boto.py", line 480, in modified_time return parse_ts(entry.last_modified) AttributeError: 'NoneType' object has no attribute 'last_modified' 

Only when I came across this solution did I start looking for what worked for me. Here the repository in which I ended up using it, saved the file locally, as well as in S3, which gave me all the errors:

 from django.contrib.staticfiles.storage import ManifestFilesMixin from django.core.files.storage import get_storage_class from pipeline.storage import PipelineMixin from storages.backends.s3boto import S3BotoStorage class StaticStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage): """Custom storage for static content.""" def __init__(self, *args, **kwargs): super(StaticStorage, self).__init__(*args, **kwargs) self.local_storage = get_storage_class( 'django.contrib.staticfiles.storage.StaticFilesStorage')() def save(self, name, content): name = super(StaticStorage, self).save(name, content) self.local_storage._save(name, content) return name