How to develop: avoid resource leaks when accidentally accessing files

I have a client / server application in which a client application will open files. These files are broken into pieces and sent to the server.

The client not only sends pieces of files, but also sends other data. Each message (data or filechunk) has a priority level.

All messages are buffered and written to the TCP stream according to their priority. Currently, input and output streams are closed on both sides (client / server) whenever the file is completely sent or received. This means that streams remain open as long as you want to send files.

The TCP connection is allowed to fail, as it will reconnect and sending the message will resume, so the streams will be closed at some point.

BUT, but if and when the JVM is killed, for example, threads will not be cleared.

My first thought about this was to add the cleanup code to the finalizers, but I understand that they may not work when the JVM is killed (or System.exit is called).

My second thought is to rewrite part of the application and use only streams as long as it takes to read / write one piece. So I would end up opening / closing files as many times as there are pieces. The advantage of this method is that I can use the try-catch-finally construct, but I have the feeling that you are opening and closing files, which often implies a fair portion of the overhead.

So, how to clear resources when the design does not allow the use of {} blocks? Or should such a construction be avoided, perhaps in the same way as I described?

I am also concerned about the possibility of having as many files as possible, as there are priorities (which are theoretically unlimited).

Most files will usually have a few KiB large, but in some cases they can reach several GB.

Thank you in advance for your entry.

EDIT: added image

File transfer as it is

+4
source share
2 answers

If I understand the question correctly, you are worried that you will not clean it properly if the JVM is terminated in an uncontrolled manner.

Although this is a common problem, in your particular case, I think there is nothing to worry about. You only open your files for reading, so there is no constant state that your application can ruin (as I understand it, on the other hand, TCP connections can correctly handle disconnections). The opened file is a kind of state that is internal to the application. If the application is killed, the operating system cleans up all locks or any data structures that it can use internally to process operations on this file. This means that there will be no โ€œgarbageโ€ in the OS kernel, and although it is neither an elegant nor recommended way to clean it, it just works (of course, use this only for emergencies, and not as a normal way of handling things). Killing your application automatically closes open files and should not bring the files into any inconsistent state.

Of course, if the application is killed, it will not complete any operations. This can lead to an inconsistent state at the application level or other problems with some types of application logic, but it still can not damage any innovations at the OS level (provided that the OS is not buggy). You can lose data or break the state of your application, but you cannot break the file system or data in the kernel.

Thus, you may encounter problems if you kill an application in which you are working in transaction mode (one that you want to fully or completely execute, but no intermediate state should become visible to the outside world), for example, if you had a file, and you had to replace it with a newer version. If you truncate the old file first and then write new content to it (which is the obvious way to do this), if your application is killed after trimming the file, but you have problems writing new content. However, in order to provoke such risks, you need to change the state, that is, write something. If you are only reading the material, you will almost certainly be safe.

If you are faced with such a case, you can do several ways. One is trying to make the application bulletproof and ensure that it always cleans well. In practice, it is very difficult. You can add termination hooks to the Java application, which will be executed when the application closes, but it only works for controlled shutdown (like regular kill (SIGTERM) on Linux). But this will not protect you from an application that will be forcibly killed by the administrator ( kill -9 on Linux), OOM-killer (also Linux), etc. Of course, other operating systems have equivalents to these situations or other cases where the application closes in such a way that it cannot be controlled. If you are not writing a real-time application running in a controlled hardware and software environment, it is almost impossible to prevent all application attempts to force-quit and prevent its cleaning procedures from being performed.

Thus, a reasonable compromise often takes only simple precautions in the application (for example, turning off hooks), but remember that you cannot prevent everything and therefore make manual cleaning possible and easy. For example, a solution to overwrite a file would be to split the operation by first moving the old file to a new name to save as a backup (this operation is usually atomic at the OS level and therefore safe), and then writing the new file contents to the old name, and then After verifying that the new file was correctly written, delete the backup. Thus, if the application is killed between operations, there is a simple cleaning procedure: transferring the backup file to its original name and, thus, returning to an older but consistent state. This can be done manually (but must be documented), or you can add a cleanup script or a special "cleanup" command to your application to make this operation simple, visible and remove the possibility of human error during the procedure. Assuming your application is rarely killed, it is usually more efficient than spending a lot of time trying to make the application bulletproof just to realize that it is really impossible. Deceiving a denial is often better than trying to prevent it (not just in programming).

You can still burn out on the OS and hardware level, but it is really difficult to prevent with the help of commercial equipment and the OS. Even if your application sees the new file in the right place, it may not have been physically written to disk, and if it is located only in the cache, it will not be lost if your application is killed, but it will be lost if someone pulls up power connect the car. But dealing with such a failure is a completely different story.

In short, in your particular case, when you are only reading files, the OS should do all the cleanup if your application is killed, and in other cases some tips are mentioned above.

+2
source

Take a look at java.lang.Runtime.addShutdownHook() . I would try adding a hook that closes all open threads, a list of which you must maintain for this case. However, pay attention to this:

In rare cases, a virtual machine can interrupt work, that is, stop working without turning it off. This happens when the virtual machine shuts down from the outside, for example, with a SIGKILL signal on Unix or a TerminateProcess call on Microsoft Windows. In addition, the virtual machine may interrupt if the native method goes awry, for example, corrupting internal data structures or trying to access non-existent memory. If the virtual machine is interrupted, then there can be no guarantee as to whether any shutdowns will be performed.

+1
source

Source: https://habr.com/ru/post/1403223/


All Articles