What is the overhead for opening a file in Windows?

Question

TL DR

How much memory does a file open in a modern Windows system? Some application loads will need to open a lot of files. Windows is very capable of opening "many" files, but what is the burden of saving one file open, so it can be decided when "many" are "too many"?

Background

For sequential processing of large data arrays (100 MB ~ several GB) inside a 32-bit process, we need to create a buffer that stores its contents on disk, and not in memory.

We have allocated a small class without any problems (using CreateFilewith FILE_ATTRIBUTE_TEMPORARYand FILE_FLAG_DELETE_ON_CLOSE).

The problem is that the way to use these buffers is such that each buffer (each temporary file) can store from several bytes to several GB of data, and we would like to keep the buffer class as minimal and maximally general.

The use case varies from 100 buffers with ~ 100 MB each to 100,000 with just a few bytes of buffers. (And yes, it is important that each buffer in this sense has its own file.)

It would seem natural to include a buffer threshold in a buffer class that is just starting to create and use a temporary file on disk, when it actually stores more bytes than the overhead (memory) when creating + links to use a temporary file - as well as the load on physical memory devices.

Question

How much memory in bytes does a temporary file open on a modern Windows system?

• CreateFile FILE_ATTRIBUTE_TEMPORARY FILE_FLAG_DELETE_ON_CLOSE
• (32-) ,
• ( )

, , ( , ) , ?

:

, , CreateFile, API- MS CRT. ( 10.00s CreateFile - - , .

: GB 32- , 2GB 32- . . .

http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx - , HANDLE 16 64- , .

STXXL, , , .

---

:

Raymond : " , , - ."

qwm : " . , - . , , _FILE_OBJECT ( _OBJECT_HEADER) ~ 300b, ."

Damon wri tes: " : 10 ( Windows 7). , , , ( MEMORYSTATUSEX::ullAvailVirtual 100 . , ). 8 16 , . 17 , 100,030 , . 412 . , 1, 60% . (...)"

" - ( , - !), CreateFile. 17 100 . 400 000 . , 10 , .

Answer 1

:

• RAM- 2G, NTFS.
• 1M (1 000 000) perfmon.

( ) :

HANDLE CreateNewTempFile(LPCTSTR filePath) {
    return ::CreateFile(
        filePath, 
        GENERIC_READ | GENERIC_WRITE, // reading and writing
        FILE_SHARE_READ, // Note: FILE_FLAG_DELETE_ON_CLOSE will also block readers, unless they specify FILE_SHARE_DELETE 
        /*Security:*/NULL, 
        CREATE_NEW, // only create if does not exist
        FILE_ATTRIBUTE_TEMPORARY | // optimize access for temporary file
        FILE_FLAG_DELETE_ON_CLOSE, // delete once the last handle has been closed
        NULL);
}

:

• , , RAM- :
  • 2060
  • 1063
  • 997
• ( ) (temp):
  • / - . 4k ( : , 10
  • /, - . 3k
  • / Nonpages Bytes - . 2,2 .
• , ( Process/Working Set).

, , ( , 16 4 ).