Why does one file object work, but not another?

Question

I wanted a file object that is dumped directly to the file when data is being written, and wrote this:

class FlushingFileObject(file): def write(self,*args,**kwargs): return_val= file.write(self,*args,**kwargs) self.flush() return return_val def writelines(self,*args,**kwargs): return_val= file.writelines(self,*args,**kwargs) self.flush() return return_val 

but it is interesting that I am not writing, but I have tried several things, including this:

 class FlushingFileObject(object): def __init__(self,*args,**kwargs): self.file_object= file(*args,**kwargs) def __getattr__(self, item): return getattr(self.file_object,item) def write(self,*args,**kwargs): return_val= self.file_object.write(*args,**kwargs) self.file_object.flush() return return_val def writelines(self,*args,**kwargs): return_val= self.file_object.writelines(*args,**kwargs) self.file_object.flush() return return_val 

which performs flash.

Why does the subclassification of file not work in this case?

Answer 1

Great question.

This is because Python optimizes calls to write on file objects by traversing the write method at the Python level and calling fputs directly.

To see this in action, consider:

 $ cat file_subclass.py import sys class FileSubclass(file): def write(self, *a, **kw): raise Exception("write called!") writelines = write sys.stdout = FileSubclass("/dev/null", "w") print "foo" sys.stderr.write("print succeeded!\n") $ python print_magic.py print succeeded! 

The write method has never been called!

Now that the object is not a subclass of file , everything works as expected:

 $ cat object_subclass.py import sys class ObjectSubclass(object): def __init__(self): pass def write(self, *a, **kw): raise Exception("write called!") writelines = write sys.stdout = ObjectSubclass() print "foo" sys.stderr.write("print succeeded!\n") $ python object_subclass.py Traceback (most recent call last): File "x.py", line 13, in <module> print "foo" File "x.py", line 8, in write raise Exception("write called!") Exception: write called! 

Digging through the Python source a little, it seems that the culprit is the PyFile_WriteString function, called by the print statement, which checks if the object being written is an instance of file , and if so, it bypasses the methods of the object and calls fputs directly:

 int PyFile_WriteString(const char *s, PyObject *f) { if (f == NULL) { /* … snip … */ } else if (PyFile_Check(f)) { //-- `isinstance(f, file)` PyFileObject *fobj = (PyFileObject *) f; FILE *fp = PyFile_AsFile(f); if (fp == NULL) { err_closed(); return -1; } FILE_BEGIN_ALLOW_THREADS(fobj) fputs(s, fp); //-- fputs, bypassing the Python object entirely FILE_END_ALLOW_THREADS(fobj) return 0; } else if (!PyErr_Occurred()) { PyObject *v = PyString_FromString(s); int err; if (v == NULL) return -1; err = PyFile_WriteObject(v, f, Py_PRINT_RAW); Py_DECREF(v); return err; } else return -1; } 

Answer 2

The file.flush() documentation says:

Note flush () does not necessarily write file data to disk. Use flush () followed by os.fsync () to provide this behavior.

I tested the first version of FlushingFileObject without calling os.fsync. The file is not cleared. After inserting os.fsync(self.fileno()) file turned red. Then I deleted the os.fsync call and now the file turns red! I think os.fsync call is required.