What is the best way to write to more files than the kernel allows you to open at the same time?

Question

I have a very large binary and I need to create separate files based on the identifier inside the input file. There are 146 output files, and I use cstdlibboth fopenand fwrite. FOPEN_MAXis 20, so I cannot open all 146 output files at the same time. I also want to minimize the number of openings and closures of the output file.

How can I efficiently write output files?

I also have to use the library cstdlibdue to legacy code.

The executable must also be compatible with UNIX and Windows.

Answer 1

A few possible approaches you can take are:

• keep a cache of open output file descriptors that are smaller than FOPEN_MAX - if recording should occur in already open files, then just write. Otherwise, close one of the pens in the cache and open the output file. If your data is usually grouped in terms of data for a specific set of files, grouped in the input file, this should work well with the LRU policy for the file descriptor cache.

• , , : 146 ( ) , open/flush/, . , /.

, , .

Answer 2

, , file1, file2 ..

Answer 3

max FOPEN_MAX - , , .

.

Answer 4

- N , N FOPEN_MAX. N . , .

Answer 5

, , . , . , thomask, . , , , .

Answer 6

, . .

, , , , . / , . , , / , .

Answer 7

2 .

1) 19 , 19 .. , 8 ( ).

2) 19 ( 13 ) .

, ID- , 163 . , , .

, .

Answer 8

- , , . , , . , .

std::map std::vector FILE. map . , vector, . . .

. , 10, 11- .

- . , ( ), ( fwrite), . , , . , . - , ( , ). , .

Answer 9

, OP, 146 , / .

, .

, . - , . - , . , 8 , 19-20 .

( ).

- . , ? ? ? "" ? ...

- . , , ? / .

Answer 10

, * nix, , . , , , . FOPEN_MAX .

, , .

Answer 11

" ":

, . , , .

:

• , .
• fseek() .
• 19 :
  • .
  • , .
  • 19 .
  • fseek() .

, . , . , .

" "

, . (). , :

• .
• , . , :
  • .
  • , fwrite(). , , (, ).
  • .

. , 200 , ?

:

• ? , " ".
• , fwrite() . , .