About calling too many open files using java?

Question

When viewing the peer code found below the code

BufferedReader br = new BufferedReader(new FileReader(PATH + fileName)); //... 

just read the file and concatenate these lines as one line, but I did not find any closed code. Therefore, I think that this should cause a resource leak and, finally, cause too many open files error , therefore, to prove this, I write a test

 for (int i = 0; i < 7168; i++) { // ulimit -n ==> 7168 BufferedReader br = new BufferedReader(new FileReader("src/main/resources/privateKey/foo.pem")); System.out.println(br.readLine()); } System.in.read(); 

Very strange, everything is in order, does not generate the expected exception.

And check the real open files on the command line

 ➜ ~ lsof -p 16276 | grep 'foo.pem' | wc -l 2538 

why only 2538 and not 7168?

So what happened? how to call too many open files error ?

---

As @GhostCat suggested, change 7168 -> Integer.MAX_VALUE, this time it called

 java.io.FileNotFoundException: src/main/resources/privateKey/foo.pem (Too many open files in system) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) 

when i'm 27436 , in which case check for real open files on the command line

 ➜ ~ lsof | grep foo.pem | wc -l 7275 

but where are the left files (27346 - 7275)? and why is the ulimit number not working?

Answer 1

I assume that the garbage collector works by finding many unreachable BufferedReader objects and collecting them. This completes the creation of the objects of the underlying thread ... that closes them.

To break this code, add BufferedReader objects to the list so that they remain available.

---

And that’s why I think changing 7168 to MAXINT works.

When the JVM starts, it will use a relatively small heap. One of the things that happens during the GC is that the JVM decides if heap size needs to be changed. So here is what could happen:

• The JVM starts with a heap that is too small to hold 7168 open files + BufferedReader objects. (Remember that each of the latter probably has a pre-allocated buffer!)

• You start to open files.

• At approximately N = 7168 - 2538, the heap fills all BufferedReader objects + FileInputStream objects + various fragments from the JVM startup / warmup.

• The GC starts up and forces (possibly) all BufferedReader objects to collect / terminate / close.

• The GC then decides that it needs to deploy the heap. You now have enough heap space for more open BufferedReader objects than your ulimit allows.

• You resume opening files ... and then delete the limit of the open file.

This is one of the possible patterns.

---

If you really want to learn this, I suggest you turn on the GC log and see if you can match the number of FDs reported by lsof to GC runs.

(You can try adding sleep calls between each open to make it easier to get lsof measurements, but this can change the behavior of the JVM in other ways ...)

Answer 2

I do not have an exact explanation, but some additional thoughts: “we” should understand that everything is not so simple as they look on the surface.

The fact is that several levels of abstraction come into play. There are JVM and JIT; and then there is an operating system below these.

Meaning: Given these abstractions, it is just too naive to expect that each new BufferReader will directly lead to a different file descriptor. It would not surprise me if the Linux kernel appeared here; and just "tells" the JVM that "yes, I opened this file and read it for you, here is its contents." But in "reality", the Linux kernel understands that this file has not been affected and has not changed since the last read request ...

Answer 3

• jvm implicitly updates ulimit value

   String [] cmdArray = {"sh","-c","ulimit -n"}; Process p = Runtime.getRuntime().exec(cmdArray); BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream())); System.out.println(in.readLine()); //it is 10240 not 7168 

• @Stephen C is right, GC is involved.

I create MyBufferedReader extends BufferedRead and overrides finalize method

 @Override protected void finalize() throws Throwable { System.out.printf("Thread: %s finalize it and total: %d %n",Thread.currentThread().getName(),count.getAndAdd(1)); } 

Get the information below.

 Thread: Finalizer finalize it and total: 9410 

and on the command line

 ➜ ~ lsof -p 5309 | grep 'taicredit_private_key_pkcs8' | wc -l 830 

and 9410 + 830 = 10240