Fortran: wait for a file to be closed by another application

Question

I have a fortran code that needs to read a series of ascii data files (which together make up about 25 GB). Basically, the code opens this ascii file, reads the information and uses it to perform some operations, and then closes it. Then another file opens, reads the information, performs some operations and closes it again. And so on with the rest of the ascii files.

In general, each full run takes about 10 hours. I usually have to run several independent calculations with different parameters, and the way I do it is to execute each independent calculation sequentially, so in the end, if I have 10 independent calculations, the total processor time is 100 hours.

A faster way would be to start 10 independent calculations simultaneously using different processors on the cluster machine, but the problem is that if this calculation should open and read data from a given ascii file that was already open and used by another calculation, then the code clearly shows a mistake.

I wonder if there is a way to check if a given ascii file is already used by another calculation, and if so, ask the code to wait until the ascii file is finally closed.

Any help would be very helpful. Thank you very much in advance.

Obamakoak.

Answer 1

Two processes should be able to read the same file. Perhaps action="read" in the statement that opens may help. Should files be human readable? I / O will probably be much faster with unformatted (sometimes binary) files.

PS If your OS does not support multi-read access, you may have to create your own locking system. Create a master file that the process opens to check which files are used or not, and update the specified list. Closes immediately after checking or updating. To handle conflicts in this read / write file, use iostat in the open statement and try again after a delay if there is an error.

Answer 2

I know this is an old thread, but I was struggling with the same problem for my own code.

My first attempt was to create a variable for a specific process (for example, a wizard) and access this variable exclusively using a one-way passive MPI. This is fantasy and works well, but only with newer versions of MPI.

Also, my code seemed happy to open (with READWRITE files) that were also opened in other processes.

Therefore, the easiest workaround, if your program has access to files, is to use an external lock file, as described here . In your case, the code might look something like this:

• The process checks for the presence of the lock file using the NEW instruction, which does not execute if the file already exists. It will look something like this:

   file_exists = .true. do while (file_exists) open(STATUS='NEW',unit=11,file=lock_file_name,iostat=open_stat) if (open_stat.eq.0) then file_exists = .false. open(STATUS='OLD',ACTION=READWRITE',unit=12,file=data_file_name,iostat=ierr) if (ierr.ne.0) stop else call sleep(1) end if end do 

• Now the file is opened exclusively by the current process. Perform the operations that you need to perform, for example, reading, writing.

• When you are done, close the data file and finally the lock file

   close(12,iostat=ierr) if (ierr.ne.0) stop close(11,status='DELETE',iostat=ierr) if (ierr.ne.0) stop 

• The data file is now unlocked again for other processes.

I hope this can be useful for other people who have the same problem.