Increasing the maximum number of open file descriptors in Matlab on Windows using fopen

Question

I have a program that should contain about 3,000 open file descriptors in Matlab. The reason for this is that if I do not open them, I need to open and close them more than 100,000 times, which means 300 million open closed operations. Given that each file is added to each time, and fopen and fclose can take more than a second (files are large, i.e. 100mb +), it should be clear that this situation is unacceptable.

I know that the file limit for Windows is set to 10000, but Matlab refuses to open more than 512 files with fopen. I can’t figure out how to make him increase this number.

Does anyone know how to change the limit of 512? Where is this defined? Is it even Matlab?

Answer 1

Can you really view your program and structure it differently to work only with a partial representation of the contents of the file contents?

For example, if you want to add 100,000 lines to 3,000 files (you don’t even need to have any idea of ​​what the files already have), you can do this as follows:

%% Main processing function [] FullProcess() %[ for block = 1:100, % Partial processing lines = processBlock(block); % Save step pushToFiles(block, lines); end %] 

FROM

 % Partial processing in memory function [lines] = processBlock(block) %[ % Preallocate lines = cells(1000, 3000); % Do the processing for current block ... lines{500, 12} = 'kikou'; ... %] 

and

 %% Save partial work function [] = pushToFiles(block, lines) %[ fcount = size(lines, 2); lcount = size(lines, 1); for fi = 1:fcount, [fid, msg] = fopen(fprintf('f%i', fi), 'a'); % Open in append mode if (fid < 0), error(msg); end for li = 1:lcount, fprintf(fid, lines{li, fi}); end fclose(fid); end %] 

This reduces the number of cases to do 100 fopen / fclose (by 3000 files, although this is much smaller than previously expected)

Answer 2

FWIW, below, this is some code to reproduce this problem:

 fids = zeros(1,513); for ix = 1:length(fids) fids(ix) = fopen(sprintf('testfile_%03d.tmp',ix),'w'); end fids(507:end) 

(After this, basic commands such as "help" fail, you need to run fclose all ).

A little web search attracts other people (in the worst Q&A forums) with the same problems, but without simple solutions (for example, this is a Mathworks forum post .)

---

My first instinct, when I encounter the limitations of Matlab, is to always turn to Java. For instance:

 streamwriters = cell(1,513); for ix = 1:length(streamwriters) strName = sprintf('testfile_2_%03d.tmp',ix); streamwriters{ix} = java.io.FileOutputStream(strName); end streamwriters{513}.write(uint8('Some data to write')) 

When you make a Java call from Matlab, you are standing (I think for a few ms), so you really do 1,000,000 entries, I would profile your code and look for ways to collect the code in memory and execute smaller, larger batches if necessary.

Also remember that you need to close them separately, for example.

 for ix = 1:length(streamwriters) streamwriters{ix}.close(); end