Improving mmap / munmap performance on MacOS X

Question

Running the following C code (a bunch of mmaps and munmaps in a 2GB file) on a MacOS X machine seems much slower than on a Linux server.

#define BUFSZ 2000000000
static u_char buf[BUFSZ];
....

// Time 10000 mmaps and munmaps from random offsets for various 
// sizes of mapped chunk.
for (msize = 4096; msize <= 1048576; msize *= 16) {
  fd = open("io_benchmark.dat", O_RDONLY);
  if (fd  < 0 ) die("can't open io_benchmark.dat for reading");
  for (i = 0; i < 10000; i++) {
    // Make sure the block to be mapped doesn't start in the
    // last meg.
    offset = (size_t) random() % (BUFSZ - 1048576);
    mblock = femmap(fd, (off_t)offset, (size_t) msize, PROT_READ, 
                    "test block");
    total = 0;
    for (j = 0; j < msize; j++) {
      total += mblock[j];
    }
    femunmap(mblock, (size_t) msize, "test block");
  }
  printf("Elapsed time to mmap and munmap 10000 blocks of %d kB: %.4f sec\n", 
         msize/1024, (time = time_since_last_call()));

  rslt = close(fd);
  if (fd  < 0 ) die("can't close io_benchmark.dat after reading");
}

In particular, a comparison of two cars

CPU     Xeon E3113 dual core @ 3.00GHz           Core 2 Duo @ 2.4GHz dual core
RAM     8GB                                      4GB
Kernel  2.6.18-92.el5PAE SMP i686                MacOS 10.6.4 Snow Leopard
Disk    WD 250GB SATA 16MB cache 7200 RPM EXT3   Hitachi 250GB SATA 5400 RPM, journaled HFS+

It gives the following results:

                            Linux    MacOS X
Time for 10000 4kB mmaps    0.0165   682.87
Time for 10000 64kB mmap    0.0170   657.79
Time for 10000 1MB mmaps    0.0217   633.38

Even considering the reduced memory size, it seems that this is unusual, since the file is only half the physical memory. Can someone point out a code change or a configuration change that could improve performance?

We are trying to use reads instead of mmaps, and this is significant, but this will require a substantial change to the existing code base (and mmap is much faster than reading on Linux).

Answer 1

, . , gcc .

, , mblock volatile. , , .

, , :

• MacOS X
• , benchmark

, , .

Answer 2

, " ". https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/FileSystemAdvancedPT/MappingFilesIntoMemory/MappingFilesIntoMemory.html#//apple_ref/doc/uid/TP40010765-CH2-SW1:

:

• .

• . ( . , , - .)

. . " ".

---

, :

off_t       file_target = off_t(3) << 29; //  1.5GB
const char* path = "B9361194.data";

//  Touch the output file
{
    FILE*       fp = fopen( path, "a");
    fclose(fp);
}

//  Open the output file
FILE*       fp = fopen( path, "rb+");
int         fd = fileno(fp);
off_t       file_physical = 0;
off_t       file_logical = 0;
ftruncate( fd, file_physical );

//  Declare the mapping
off_t       map_start = 0;
off_t       map_count = 0;
char*       map_address = 0;

//  Set up the input buffer.
//  We are just going to write this out until we have written total bytes
size_t      requested = 1024;
char    input[requested];
for ( size_t i = 0; i < requested; ++i ) input[i] = 1;

//  Write the buffer, resizing and mapping as we go.
while ( file_logical < file_target ) {
    //  Figure out how much to write.
    size_t limit = requested;
    if ( ( file_target - file_logical ) < (off_t)limit ) 
        limit = size_t( file_target - file_logical );

    //  If we can't fit the buffer inside the allocation
    //  unmap and grow everything
    if ( file_logical + (off_t)limit > file_physical ) {
        //  Unmap
        if ( map_address ) munmap( map_address, map_count );

        //  Grow the file by 64K
        off_t   new_physical = off_t(1) << 16;  //  64K allocation
        if ( new_physical < (off_t)limit ) new_physical = limit;
        file_physical += new_physical;
        ftruncate( fd, file_physical );

        //  Map the end
        map_count = off_t(1) << 23;    //  8MB
        if ( map_count > file_physical ) map_count = file_physical;
        map_start = file_physical - map_count;
        void* address = mmap( 0, map_count, ( PROT_WRITE | PROT_READ ), MAP_SHARED, fd, map_start );
        // int err = errno;
        // if ( address == MAP_FAILED ) CPPUNIT_ASSERT_EQUAL_MESSAGE( strerror(err), 0, errno );
        map_address = reinterpret_cast<char*>( address );
    }

    //  Copy the buffer in
    size_t  offset = size_t(file_logical - map_start);
    memcpy( map_address + offset, input, limit );
    file_logical += limit;
}

//  Clean up
if ( map_address ) munmap( map_address, map_count );
ftruncate( fd, file_logical );
fclose( fp );

, .

Answer 3

My conclusion to the OS X mmap problem is to display the entire file and save it. If you need to expand the file, draw more bytes than you need, so you only need to redo it from time to time.

You may need to use the 64-bit OS X to make this work.