Check if buffers have been flushed in R

Question

I have large, large files that I work with, and I use several different I / O functions to access them. The most common is the bigmemory package.

When writing to files, I learned a hard way to flush output buffers, otherwise all bets will not be saved. However, this can lead to a very long wait time, while the bigmemory does its thing (many minutes). I do not know why this happens - it does not always happen, and it is not easy to reproduce.

Is there a way to determine if I / O buffers in R have been bigmemory , especially for bigmemory ? If the operating system matters, then feel free to hold back the answer this way.

If the answer can be generalized outside of bigmemory , that would be great, as I sometimes rely on other memory mapping functions or I / O streams.

If there are no good solutions for checking the flush of buffers, are there cases where it can be assumed that the buffers were flushed? That is, in addition, using flush() .

Update: I have to clarify that these are all binary connections. @RichieCotton noted that isIncomplete() , although only textual connections are mentioned in the help documentation. It is unclear whether this can be used for binary connections.

Answer 1

I will state my own answer, but I welcome all that is clearer.

From what I have seen so far, various connection functions, for example. file , open , close , flush , isOpen and isIncomplete (among others) are based on specific connection types, e.g. files, feeds, URLs and some other things.

In contrast, bigmemory has its own type of connection, and the bigmemory object is an S4 object with a slot for a memory address for operating system buffers. After being placed there, the OS is responsible for flushing these buffers. Since this is the responsibility of the OS, obtaining information about dirty buffers requires interaction with the OS, not R.

Thus, the answer for bigmemory is “no,” because the data is stored in the kernel buffer, although it may be “yes” for other connections that are processed through STDIO (that is, stored in “user space”).

For more information on OS / kernel actions, see this question in SO ; I am studying a couple of programs (and not just R + bigmemory) that produce buffering that fills oddities, and this thread helped enlighten me on the kernel side.

Answer 2

How convincingly is isIncomplete () working with binary files?

 # R process 1 zz <- file("~/test", "wb") writeBin(c(1:100000),con=zz) close(zz) # R process 2 zz2 <- file("~/test", "rb") inpp <- readBin(con=zz2, integer(), 10000) while(isIncomplete(con2)) {Sys.sleep(1); inpp <- c(inpp, readBin(zz2),integer(), 10000)} close(zz2) 

(Changed from the help file (connections).)