Is 64K fully used for every pipe created?

Question

How are buffers executed? I could create many channels, but only ever sent / received several bytes through them at a time, so I do not want to waste memory unnecessarily.

Edit: I understand what buffering is, I ask how buffering is implemented in Linux channels specifically, i.e. Does it get full 64K regardless of highwatermark?

Answer 1

Buffers are used to equalize the difference in speed between producer and consumer. If you didn’t have a buffer, you would have to switch tasks after each byte created, which would be very inefficient due to the cost of context switches, data and code caches never got hot, etc. If your consumer can create data as fast as the manufacturer consumes, your buffer usage will usually be low (but read on). If the producer is much faster than the consumer, the buffer will be full, and the producer will have to wait until there is more free space. The reverse case of a slow producer and a fast consumer will use most of the buffer most of the time.

The use also depends on whether your both processes will actually be executed in parallel (for example, on separate kernels) or if they share the kernel, and only because of process control the OS are deceived into thinking that they are parallel. If you have real concurrency (separate core / processor), your buffer will usually be used less.

In any case, if your applications do not produce a lot of data, and their speeds are similar, the buffer will not fill up most of the time. Nevertheless, I would not be surprised if at the OS level the full 64 kB was allocated in any way. But if you do not use the built-in device, 64 KB is not so much, so even if there is always a maximum size, we would not worry about that.

By the way, changing the size of the channel buffer, for example, in this discussion, is not easy, several tricks are suggested, but they are actually workarounds that change the way data is used from the buffer, and not change the size of the actual buffer. You can check ulimit -p , but I'm not 100% sure that it will give you the control you need.

EDIT : look at fs/pipe.c and include/linux/pipe_fs_i.h in Linux code, it looks like buffers are include/linux/pipe_fs_i.h . However, the minimum buffer size is a full page, so if you need only a few bytes, there will be waste. I'm not sure at the moment, but some code that uses PIPE_DEF_BUFFERS , which is 16, giving 64 KB with 4 KB pages, makes me wonder if the buffer can drop below 64 KB (at least 1 page may just be an additional restriction).