Creating Unfixed File Output Buffers

Question

I am trying to fix a problem that occurs when using unsecured file I / O buffers in several programs in different languages ​​running on Linux. The solution for flushing buffers is quite simple, but the question of non-fading buffers is rather random. Instead of asking for help on what might cause this, I'm interested in how to create (reproduce) and diagnose a similar situation.

This leads to a two-part question:

• Is it possible to artificially and easily build instances in which output buffers, which, as you know, are not unlocked, can be created over a certain period of time? My searches are empty. The trivial baseline is to clog a hard drive (e.g. swap) in one process, trying to write a large amount of data from another process. Although this "works", it makes the system almost unusable: I can not poke and see what happens.

• Are there any Linux commands that can identify that a given process has loose output buffers? Is this something that can be run on the command line or you need to query the kernel directly? I watched fsync , sync , ioctl , flush , bdflush and others. However, without a method for creating unbound buffers, it is unclear what they might reveal.

To reproduce for others, the example for # 1 in C would be excellent, but the question is really linguistic agnostic - just knowing the approach to creating this situation will help in other languages ​​in which I work.

---

Update 1: I apologize for any confusion. As several people have noted, buffers can be in kernel space or in user space. This helped identify problems: we create large dirty kernel buffers. This distinction and answers fully resolve No. 1: it now seems clear how to recreate unexpanded buffers in user space or in kernel space. The identification of a process identifier with dirty kernel buffers is not yet clear.

Answer 1

If you are interested in kernel buffered data, you can configure VM writeback through sysctls in /proc/sys/vm/dirty_* . In particular, dirty_expire_centisecs is the age in hundredths of a second at which dirty data becomes dirty_expire_centisecs . Increasing this value will give you more time to conduct your investigation. You can also increase dirty_ratio and dirty_background_ratio (which are percent of system memory, defining the point at which synchronous and asynchronous recording starts).

In fact, creating dirty pages is easy - just write(2) to the file and exit without synchronization, or contaminate some pages in the MAP_SHARED file association.

Answer 2

A simple program that will have a loose buffer will be:

 main() { printf("moo"); pause(); } 

Stdio, by default only resets stdout in new lines when connected to the terminal.

Answer 3

It is very easy to call unmanaged buffers by monitoring the receiving side. The beauty of * nix systems is that everything looks like a file, so you can use special files to do what you want. The easiest option is a pipe. If you just want to control stdout, this is an option: unflushed_program | slow_consumer unflushed_program | slow_consumer Otherwise, you can use named pipes:

 mkfifo pipe_file unflushed_program --output pipe_file slow_consumer --input pipe_file 

slow_consumer is most likely a program that you develop to read data slowly, or just read the X bytes and stop.