I am trying to use a memory mapped file to write an application with high I / O requirements. In this application, I have a data packet that arrives at a speed exceeding the speed that the disk can support. To avoid buffering logic in my application, I was thinking about using a memory mapped file. With such a file, I would simply write in memory that is mapped to a file (faster than what disks can support), and the OS will eventually clear this data on disk. Therefore, the OS is buffering for me.
After the experiment, I see that files with memory mapping speed up writing to memory, but flushing to disk is slower than with a regular file. This is what leads me to this conclusion. Here is a piece of code that simply writes as fast as it can to a memoryless file:
private static void WriteNonMemoryMappedFile(long fileSize, byte[] bufferToWrite)
{
Console.WriteLine(" ==> Non memory mapped file");
string normalFileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-NonMmf.bin");
if (File.Exists(normalFileName))
{
File.Delete(normalFileName);
}
var stopWatch = Stopwatch.StartNew();
using (var file = File.OpenWrite(normalFileName))
{
var numberOfPages = fileSize/bufferToWrite.Length;
for (int page = 0; page < numberOfPages; page++)
{
file.Write(bufferToWrite, 0, bufferToWrite.Length);
}
}
Console.WriteLine("Non-memory mapped file is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
}
This code leads to the following:
==> Non memory mapped file
Non-memory mapped file is now closed after 10.5918587 seconds (966.687541390441 MB/s)
As you can see, my drives are pretty fast. This will be my guide for memory mapped files.
Now I tried to write the same data to a memory mapped file using unsafe code (because this is what I intend to do in my application):
[DllImport("msvcrt.dll", EntryPoint = "memcpy", CallingConvention = CallingConvention.Cdecl, SetLastError = false)]
public static extern IntPtr memcpy(IntPtr dest, IntPtr src, UIntPtr count);
private static unsafe void WriteMemoryMappedFileWithUnsafeCode(long fileSize, byte[] bufferToWrite)
{
Console.WriteLine(" ==> Memory mapped file with unsafe code");
string fileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-MmfUnsafeCode.bin");
if (File.Exists(fileName))
{
File.Delete(fileName);
}
string mapName = Guid.NewGuid().ToString();
var stopWatch = Stopwatch.StartNew();
using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(fileName, FileMode.Create, mapName, fileSize, MemoryMappedFileAccess.ReadWrite))
using (var view = memoryMappedFile.CreateViewAccessor(0, fileSize, MemoryMappedFileAccess.Write))
{
unsafe
{
fixed (byte* pageToWritePointer = bufferToWrite)
{
byte* pointer = null;
try
{
view.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);
var writePointer = pointer;
var numberOfPages = fileSize/bufferToWrite.Length;
for (int page = 0; page < numberOfPages; page++)
{
memcpy((IntPtr) writePointer, (IntPtr) pageToWritePointer, (UIntPtr) bufferToWrite.Length);
writePointer += bufferToWrite.Length;
}
}
finally
{
if (pointer != null)
view.SafeMemoryMappedViewHandle.ReleasePointer();
}
}
}
Console.WriteLine("All bytes written in MMF after {0} seconds ({1} MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
}
Console.WriteLine("File is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
}
Then I get the following:
==> Memory mapped file with unsafe code
All bytes written in MMF after 6.5442406 seconds (1564.73302033172 MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.
File is now closed after 18.8873186 seconds (542.162704287661 MB/s)
As you can see, this is much slower. It writes about 56% of a non-memory file.
Then I tried another. I tried using ViewStreamAccessor instead of unsafe code:
private static unsafe void WriteMemoryMappedFileWithViewStream(long fileSize, byte[] bufferToWrite)
{
Console.WriteLine(" ==> Memory mapped file with view stream");
string fileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-MmfViewStream.bin");
if (File.Exists(fileName))
{
File.Delete(fileName);
}
string mapName = Guid.NewGuid().ToString();
var stopWatch = Stopwatch.StartNew();
using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(fileName, FileMode.Create, mapName, fileSize, MemoryMappedFileAccess.ReadWrite))
using (var viewStream = memoryMappedFile.CreateViewStream(0, fileSize, MemoryMappedFileAccess.Write))
{
var numberOfPages = fileSize / bufferToWrite.Length;
for (int page = 0; page < numberOfPages; page++)
{
viewStream.Write(bufferToWrite, 0, bufferToWrite.Length);
}
Console.WriteLine("All bytes written in MMF after {0} seconds ({1} MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
}
Console.WriteLine("File is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
}
Then I get the following:
==> Memory mapped file with view stream
All bytes written in MMF after 4.6713875 seconds (2192.06548076352 MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.
File is now closed after 16.8921666 seconds (606.198141569359 MB/s)
, , , .
, -, , ?
, :
static void Main(string[] args)
{
var bufferToWrite = Enumerable.Range(0, Environment.SystemPageSize * 256).Select(i => (byte)i).ToArray();
long fileSize = 10 * 1024 * 1024 * 1024L;
WriteNonMemoryMappedFile(fileSize, bufferToWrite);
WriteMemoryMappedFileWithUnsafeCode(fileSize, bufferToWrite);
WriteMemoryMappedFileWithViewStream(fileSize, bufferToWrite);
}
private static double GetSpeed(long fileSize, Stopwatch stopwatch)
{
var mb = fileSize / 1024.0 / 1024.0;
var mbPerSec = mb / stopwatch.Elapsed.TotalSeconds;
return mbPerSec;
}
1:
usr, SequenctialScan. , . , :
using (var file = new FileStream(fileName, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.None, 4096, FileOptions.SequentialScan))
using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(file, mapName, fileSize, MemoryMappedFileAccess.ReadWrite, null, HandleInheritability.None, leaveOpen: false))