Is there a restriction on concurrent threads?

Question

I have a strategic question about using simultaneously open fstream s. I have to write a program that read too many files too much. Each file has information for a group of identifiers, but once. I have to calculate this information and save it for each identifier in a separate file. Each identifier appears in several files and must be saved each time in one file (one identifier many times). I expect several hundred identifiers, so I doubt that I have to open several hundred files at the same time.

So, is there a restriction on concurrent threads? Or do you suggest another way to do this?

The program will calculate a huge amount of data (about 10 GB or more) and, possibly, will calculate several hours.

thanks

Answer 1

Ultimately, this is a limitation. Files are a great example of what is controlled by the operating system, and you will have to consult the documentation of your OS for a specific limit. On Linux, I believe that it is configured in the kernel. Additionally, there may be user and technological quotas.

I do not think 200 is too much to ask.

It is quite simple to try and see. Just write a program that will open more files until you get an error message.

Living example.

On Mac OS X 10.8, this program

 #include <iostream> #include <fstream> #include <iomanip> #include <string> int main() { int i = 0; std::ofstream *f; do { f = new std::ofstream( std::to_string( i ++ ) ); } while ( * f << "hello" << std::flush ); -- i; // Don't count last iteration, which failed to open anything. std::cout << i << '\n'; } 

Outputs 253 . So, if you're on a Mac, you're golden :).

Answer 2

The C ++ standard does not define a limit for how many (or how few, I believe, but I have not watched) files that you can open at a time.

In particular, a C ++ library implementation may have a limitation (which may or may not be documented). The operating system is likely to have a certain limit for the entire system and another limit for each process. What these restrictions will vary, so there is no easy way to tell. And they can also be artificially omitted by various settings that the system owner configures.

And even if you know what all these restrictions are, there may be dynamic restrictions that depend on the circumstances, for example, if the entire system allows 16384 files to be opened, 1000 for each process limit, and the C ++ library allows 1024, maybe you don’t it will be possible to open one file, since there is no free memory for the OS to allocate some critical data block.

Answer 3

There are no restrictions on the streams that you can open at the same time, however your limits limit the number of files that can be opened at the same time. Although some hundreds of files do not seem to be too large for a common OS, I would advise you to read all the information in advance (possibly opening several files at the same time, but given the possibility of calling “open” for failure, in this case you should try again after closing some previously opened files), then process and save the results in some internal data structure. Finally, you can write the results to files again, in parallel, but again, be prepared for an unsuccessful attempt to open the file.

Answer 4

• Os may impose a limit on the number of simultaneously open files. Unix-like systems (linux, * bsd, etc.) definitely have this limit and are configurable, windows can have a similar configurable limit
• On any operating system, you cannot open more than 2^(8*sizeof(filehandle)) individual files. filehandle is a type used to access the contents of a file. HANDLE, FILE *, int, etc. Depends on the operating system. You will probably run out of memory before you reach this limit.
• The Windows C runtime library (stdio, which provides the fprintf function and a similar function) can open no more than 512 files at a time, this number can be increased to 2048, but not further. See _ setmaxstdio . As a result, if fstream uses cstdio under the hood, the same limit will apply to fstream.
• People say that on 32 windows xp one process cannot open more than 65535 files. However, this information is a rumor, it does not seem to be supported by the msdn documentation. This means that this is probably not true.