IOError: there is no space on the device - which device?

Question

I upload a small file (8.5 Mb) to a test flash drive server.

When the file completes the download, the server reports:

File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/wtforms/form.py", line 212, in __call__ return type.__call__(cls, *args, **kwargs) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask_wtf/form.py", line 49, in __init__ formdata = request.form File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/local.py", line 338, in __getattr__ return getattr(self._get_current_object(), name) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/utils.py", line 71, in __get__ value = self.func(obj) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", line 484, in form self._load_form_data() File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask/wrappers.py", line 165, in _load_form_data RequestBase._load_form_data(self) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", line 356, in _load_form_data mimetype, content_length, options) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", line 193, in parse content_length, options) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", line 99, in wrapper return f(self, stream, *args, **kwargs) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", line 210, in _parse_multipart form, files = parser.parse(stream, boundary, content_length) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", line 520, in parse return self.cls(form), self.cls(files) File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/datastructures.py", line 373, in __init__ for key, value in mapping or (): File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", line 518, in <genexpr> form = (p[1] for p in formstream if p[0] == 'form') File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", line 494, in parse_parts _write(ell) IOError: [Errno 28] No space left on device 

Now the server has a lot of free space - through 3Gb.

I looked at the Werkzeug github repo to try and find the place that Werkzeug is trying to record, but cannot track it down.

I also checked tempfile.gettempdir () , which gives / var / tmp as a temporary directory of files, but this folder is almost empty, so I don’t think the place creating the problem.

How to find a device in which there is no place?

Answer 1

Comment @Tom Hunt was on the right track.

This unix SE answer explains what happened .

As a protection against small disk space, some daemons automatically “shadow” the current / tmp / dir with the disk if the root partition ends on the disk. Unfortunately, there is no automatic return to this process as soon as sufficient disk space becomes free again.

I disabled the / tmp directory and followed the Nitesh suggestion:

 sudo umount /tmp sudo echo 'MINTMPKB=0' > sudo /etc/default/mountoverflowtmp 

and now the download works correctly.

Answer 2

If you can get a shell on the server, try typing df -h and look for any entries that show Use% of 100% or Avail less than the size of your file.

Answer 3

Werkzeug stores files at a specific size in your temp directory using tempfile.TemporaryFile() , but note that these files are disabled for security. You will not see them in the list. tempfile.gettempdir() is the correct method to determine the directory used for this.

Perhaps your /var/tmp configured for a smaller partition. See df -h to see if there is enough free space in this section. You also need to check out the free inodes, df -i .

It may also be that the process (perhaps yours) has been hanging on such unrelated files for too long, and the space has not yet been returned to the OS. You can check processes stored on remote files with:

 lsof -nP | grep '/var/tmp' | grep '(deleted)' 

or

 find /proc/*/fd -ls | grep '/var/tmp' | grep '(deleted)' 

Answer 4

Try df -i , maybe there are no free indexes.

EDIT:

another option, find werkzeug pid, let it be 777,

• run strace -p 777 &> /tmp/strace_log
• try downloading the file
• stop strace
• find the message No space left on device , it will be like write(1, "blah blah ..."..., 57) = -1 ENOSPC (No space left on device) first argument to write is the file descriptor
• and try to find the specific syscall open(... = X , that X is a file descriptor that will not execute the "syscall" write step

Pretty bulky, I know, but this debugging.