This question is related to the previous question that I posted: Windows fsync (FlushFileBuffers) performance with large files . Where I found a possible solution, but also new questions.
When comparing different scenarios for phishing entries, I found a number of unexpected results. I hope someone can help explain or point me in the direction of information that explains these results.
In this recording method, random blocks (pages 4096 bytes in size) are written to the file sequentially in batches of 8 pages (32 K), and then the records are reset. He writes a total of 200,000 pages, a total of 800 MB and 25,000 flash drives. The file size is set to the final length before starting recording.
It supports a total of 4 options, of which all combinations are performed:
- To perform the operation "fsync" /
FlushFileBuffers (FS) after recording a part or a regular flash (NS). - To write one byte to the last position of the file before recording (LB) or leave the file empty (E).
- To use regular buffered recording (B) or recording without buffering / recording (WT) (using FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH).
- To write directly to a file stream, that is, through a file descriptor (F) or indirect recording to a file using a memory card (MM).
The table below shows my results on my system (64-bit Win 7-laptop with a slow spindle drive) for all combinations of these parameters.
What I discovered is that the performance of fsynced buffered records decreases exponentially with file size to incredibly low bandwidth, which makes it impossible to run in conjunction with large files. If the last byte (LB option) was written to the file, the throughput is even lower, so I fear that random rather than sequential write scenarios, the performance will be even more dramatic.
Surprisingly, however, with unbuffered / numbered I / O, throughput remains constant regardless of file size. Initially (the first 100-200 MB) it has a lower bandwidth than buffered records, but after that the average bandwidth quickly picks up speed and ends recording 800 MB much faster. Even more surprisingly, if the file had its last byte, the throughput is doubled.
When writing to a file through a file with memory mapping, the same exponential decrease in performance is observed as in the case when the file was opened with unbuffered / numbered flags. And here performance also worsens if the file has a byte written to the last position.
UPDATE Based on Howard's explanations here and here , I repeated the test without creating a new file before starting recording (i.e. opening an existing fully recorded file and overwriting it). I updated the code in my original question to reflect the changes made for this test. The results partially correspond to his explanations and conclusions in Linux. But there are some notable exceptions. The table below shows the results, red highlights significant changes, blue highlights where no changes have occurred, and this is surprising (i.e. does not meet expectations if the effects mentioned in Howard's explanation were the only ones in the game).
For buffered writing to a file (that is, not via memmap) with the "fsync" flash, the performance has now changed from exponential decay to a constant trend. However, it takes much longer than in previous test cases. The throughput is constant 1.5 MB / s, where before starting work, about 20 MB / s exponentially decreases to 1.5 MB / s. It seems like a possible explanation is that file metadata is also eroded on each stream, causing a complete disk revolution to look for metadata locations.
For write-to-file scenarios, the results for writing the last byte or not are now identical in accordance with what is expected from Howard's explanation.
However, the recordings on the memory card, with one notable exception, have not really changed, and this is surprising. They still show the same exponential decline in write performance (starting at 20 MB / s, decomposing to 1.8 MB / s). This suggests that another mechanism is playing. The only notable exception is if the main file was created without FILE_FLAG_WRITE_THROUGH and fsync resets are performed. This scenario now shows consistent (low) performance with a throughput of about 1.6 MB / s. Since I had some doubts, I repeat this scenario several times, each time getting the same result.
To find out a little more, I will also repeat this test using a smaller file (50,000 pages, 200 MB) to confirm that the fsync performance (for buffered I / O) actually depends on the file size. The results are shown below, and those that deserve special attention are highlighted in red.
These results correlate well with what was visible for the larger file. Noticeable changes are that the records are a little more efficient for those that are highlighted, where they seem to reach a limit of about 7 MB / s.
To summarize as highly speculative conclusions based on observations on my system :
- "fsync" windows performance on files with buffered IO (i.e. without FILE_FLAG_WRITE_THROUGH flags) decreases exponentially with the number of bytes already written to the file. The reason, apparently, is the need to clear the file metadata every time, which causes the disk to look for the beginning of the file.
- "fsync" performance on Windows when writing to a memory mapped file also shows exponentially decreasing performance. I currently have no explanation for the exact mechanism (s) causing this.
Given this observable performance, at least for my use case, these two I / O options will not represent feasible solutions.
According to Greg’s suggestion, I will restart the test with disabling Windows caching, and I will also run Howard provided a control code to eliminate the possibility of skewing the results due to errors in my own.
UPDATE 2 I have completed the tests and am currently compiling the results. In order not to write a “full story”, I will replace the current content of this question with a summary of the results, conclusions and some conclusions. Howard's answers to this question and the ability to run his base code with code next to .NET were most useful. The results of these applications correlate quite well. Rlb’s answer helped me better understand what “reasonable numbers” are related to disks. Thank you
Part of the question remains unanswered. This is especially due to the observed decrease (and depending on the file size) performance when writing to a memory card. This may be due to retrieval / metadata attempts, but it’s not yet clear to me why and how.