Passing a file instance as an argument to the celery task raises "ValueError: I / O operation in a closed file"

Question

I need to pass the file as an argument to the celery task, but the transferred file is somehow closed. This happens in case I execute the task asynchronously. Is this expected behavior?

Views:

from engine.tasks import s3_upload_handler def myfunc(): f = open('/app/uploads/pic.jpg', 'rb') s3_file_handler.apply_async(kwargs={"uploaded_file" : f,"file_name" : "test.jpg"}) 

tasks:

 def s3_upload_handler(uploaded_file,file_name): ... #some code for uploading to s3 

tracking:

 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/app/photohosting/engine/tasks.py", line 34, in s3_upload_handler key.set_contents_from_file(uploaded_file) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1217, in set_contents_from_file spos = fp.tell() ValueError: I/O operation on closed file 

flower magazines:

 kwargs { 'file_name': 'test.jpg', 'uploaded_file': <closed file '<uninitialized file>', mode '<uninitialized file>' at 0x7f6ab9e75e40> } 

Answer 1

Yes, of course, the file will be closed. The tasks of asynchronous celery are performed in a completely separate process (moreover, they can even work on another machine), and there is no way to transfer an open file to it.

You must close the file in the process from which you are invoking the task, and then transfer your name and possibly put it in the file (if necessary) to the task, and then open it again in the task.

Answer 2

Another way to do this is to open the file and get the binary block that you pass through the wiring. Of course, if the file is really large, then what @Vasily says is better, but it doesn’t work if the worker is working on another m / c (if your file is not on shared storage).