How to verify that a file is complete before trying to process it?

Question

It's hard to give a good headline to my problem, but it is. At first I do this on windows, and it can be used in the linux box too, so I need the fix to work on both systems. I keep track of the directory for new files. I basically look at directory files and compare them over and over and process only new files. The problem is that I keep getting an error when the file does not end, when I try to process the file.

public class LiveDetectionsProvider extends DetectionsProvider { protected LiveDetectionsProvider.MonitorDirectory monitorDirectory = null; protected TimeModel timeModel = null; private ArrayList<String> loaded = new ArrayList(); private File topLayerFolder = null; public LiveDetectionsProvider(String directory, String id) { super(directory, id); timeModel = super.timeModel; } /** * Initialize the data provider. */ public void initialize() { try { topLayerFolder = new File(directory); File[] dir = topLayerFolder.listFiles(); for (File file : dir) { loaded.add(file.getName()); } monitorDirectory = new MonitorDirectory(); monitorDirectory.execute(); } catch (Exception ex) { Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Failed to read detection\n{0}", ex.getMessage()); } super.initialize(); } /** * Un-initialize the data provider. */ public void uninitialize() { super.uninitialize(); if (monitorDirectory != null) { monitorDirectory.continuing = false; } } /** * The class that is used to load the detection points in a background * thread. */ protected class MonitorDirectory extends SwingWorker<Void, Void> { public boolean continuing = true; /** * The executor service thread pool. */ private ExecutorService executor = null; /** * The completion service that reports the completed threads. */ private CompletionService<Object> completionService = null; @Override protected Void doInBackground() throws Exception { int count = 0; executor = Executors.newFixedThreadPool(1); completionService = new ExecutorCompletionService<>(executor); while (continuing && topLayerFolder != null) { File[] dir = topLayerFolder.listFiles(); Thread.sleep(10); ArrayList<File> filesToLoad = new ArrayList(); for (File file : dir) { if (!loaded.contains(file.getName())) { long filesize = 0; boolean cont = true; while (cont) { if (file.length() == filesize) { cont = false; Thread.sleep(3); filesToLoad.add(file); } else { filesize = file.length(); Thread.sleep(3); } } Thread.sleep(3); } } for (File file : filesToLoad) { timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), true); completionService.submit(Executors.callable(new ReadDetection(file, false))); while (completionService.take() == null) { Thread.sleep(2); } loaded.add(file.getName()); count++; Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Detection Message Count:" + count); } detectionsModel.fireStateChanged(DetectionsModel.CHANGE_EVENT_DETECTIONS); timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), false); } return null; } } } 

The file is processed in line using

 completionService.submit(Executors.callable(new ReadDetection(file, false))); 

The file has not finished writing at this point and thus does not work. I tried sleeping my thread to slow it down, and I tried to check the size of the file, which did not change. My test case: I will unzip a tar file containing thousands of files of size 1000 KB.

Answer 1

First, yes, it compiles, see the solution that I posted in relation to the code. In my question. You replace

 For(File file: dir){ while(!file.renameTo(file)){ Thread.sleep(1) } // In my code I check to see if the file name is already in the list which // contains files that have been previously loaded if its not I add it to a list // of files to be processed } 

in for

  for (File file : dir) { if (!loaded.contains(file.getName())) { long filesize = 0; boolean cont = true; while (cont) { if (file.length() == filesize) { cont = false; Thread.sleep(3); filesToLoad.add(file); } else { filesize = file.length(); Thread.sleep(3); } } Thread.sleep(3); } } 

Sorry, I forgot to put comment tags // in the line that said do what you need to do here.

What it does is look at each file in the directory and check if you can rename it if the rename did not work, and continues to check until it can rename, and at what point you can do what you need with the file, which in my case was everything after the for loop that was replaced. I am curious why my tent was viewed as an optional item, deleted and blocked. This solution really works and solves my problem, and someone else who has the same problem is trying to process a file that is still being written or copied to a directory that is being monitored for changes.

Answer 2

I usually solve this problem by creating a temporary file when the file is being written. Upon completion, I will rename the file, and only the renamed file can be the process.

Answer 3

Use the "flag file": after completing the file.txt file, specify this by creating the .flg file - the consumption process should wait for .flg to appear.