"RuntimeError: fdopen () ended unexpectedly" when reading too many gml files

Question

I read too many GML files (several thousand) using igraph with python. At some point in the code run, I received the following runtime error:

RuntimeError: fdopen() failed unexpectedly 

I spent a lot of time trying to understand the reasons, but I did not find anything useful.

The main reason here was caused by the code https://github.com/igraph/python-igraph/blob/master/src/filehandle.c#L231

The code I use is below. It breaks through the GML reading line after reading only a few hundred files.

 gmls= [] for f in sorted(glob.glob('path_to_gmls'), key=os.path.getsize): g = Graph.Read_GML(f) gmls.append(g) 

UPDATE: I tried the same code on Mac and everything works fine. Problems arise in Windows.

UPDATE2: I tested the package using the following code and it works without problems.

 import igraph.test igraph.test.run_tests() 

Answer 1

The igraph code in the link swallows errno returned by fdopen() . You can show the actual error by opening the input file on the Python layer:

 gmls = [] for f in sorted(glob.glob('path_to_gmls'), key=os.path.getsize): with open(f, 'r') as infile: g = Graph.Read_GML(infile) gmls.append(g) 

Possible reasons are: glob.glob() returning the directory name among GML files, the name of the file with special characters, lack of permission, restriction of the network resource descriptor.

Answer 2

Are you running out of memory? (I am not a python programmer). If multiple files are opened at the same time, then the buffers for these open descriptors can take up a significant amount of memory. Also, it looks like you are adding the contents of a file to an array in memory. If the content is large, it could be a memory issue.