Out-of-memory archive archiving

Question

I had a problem running a console job and a daily log file is being created, which I archive at midnight.

This creates an empty log file the next day and an archive file with the date of the day in the name and contents of the old file to debug problems that may have been unknown until the next day.

However, since I completed the BOT task, I had problems with System Out of Memory errors when trying to archive the file.

At first, I simply could not get the archive file, but I developed a way to get at least the last 100,000 lines, which are not enough.

I wrap everything in 3 attempts / catches

• I / O
• Memory system
• standard exception

However, this is always an OutOfMemoryException exception that I get, for example,

Error System.OutOfMemoryException: An exception of type "System.OutOfMemoryException" was thrown .;

To give you an example of a size of 100,000 log lines, this is about 11 MB of file

The standard full log file can be any size from 1/2 to 2 GB.

I need to know the following:

a) what size of the standard text file will throw an error out of memory when trying to use File.ReadAllText or the custom StreamReader function that I call ReadFileString, for example,

public static string ReadFileString(string path) { // Use StreamReader to consume the entire text file. using (StreamReader reader = new StreamReader(path)) { return reader.ReadToEnd(); } } 

b) this is the memory of my computer (I have 16 GB of RAM - 8 GB when copying) or the objects that I use in C # that do not work when opening and copying files.

When archiving, I will first try using my special function ReadFileString (see above), if it returns 0 bytes of content, I try File.ReadAllText, and then if that fails, I try to execute a user-defined function to get the last 100,000 lines which is really not enough to debug errors earlier than a day.

The log file starts at midnight when a new log is created and recorded all day. I never had errors in memory, but since I increased the frequency of method calls, the log expanded, which also means that file sizes also have.

This is my special function to get the last 100,000 lines. I am wondering how many lines I could get without an IT throw from memory, and I did not get any contents of the recent days log file at all.

What do people recommend for the maximum file size for the various methods / memory needed to store the X strings, and what is the best method to get as many log files as possible?

EG somehow loop through line until it throws an exception, and then save what I have.

This is my GetHundredThousandLines method, and it gets logged in a very small debug file, so I can see what errors occurred during the archive process.

 private bool GetHundredThousandLines(string logpath, string archivepath) { bool success = false; int numberOfLines = 100000; if (!File.Exists(logpath)) { this.LogDebug("GetHundredThousandLines - Cannot find path " + logpath + " to archive " + numberOfLines.ToString() + " lines"); return false; } var queue = new Queue<string>(numberOfLines); using (FileStream fs = File.Open(logpath, FileMode.Open, FileAccess.Read, FileShare.Read)) using (BufferedStream bs = new BufferedStream(fs)) // May not make much difference. using (StreamReader sr = new StreamReader(bs)) { while (!sr.EndOfStream) { if (queue.Count == numberOfLines) { queue.Dequeue(); } queue.Enqueue(sr.ReadLine() + "\r\n"); } } // The queue now has our set of lines. So print to console, save to another file, etc. try { do { File.AppendAllText(archivepath, queue.Dequeue(), Encoding.UTF8); } while (queue.Count > 0); } catch (IOException exception) { this.LogDebug("GetHundredThousandLines - I/O Error accessing daily log file with ReadFileString: " + exception.Message.ToString()); } catch (System.OutOfMemoryException exception) { this.LogDebug("GetHundredThousandLines - Out of Memory Error accessing daily log file with ReadFileString: " + exception.Message.ToString()); } catch (Exception exception) { this.LogDebug("GetHundredThousandLines - Exception accessing daily log file with ReadFileString: " + exception.Message.ToString()); } if (File.Exists(archivepath)) { this.LogDebug("GetHundredThousandLines - Log file exists at " + archivepath); success = true; } else { this.LogDebug("GetHundredThousandLines - Log file DOES NOT exist at " + archivepath); } return success; } 

Any help would be greatly appreciated.

thanks

Answer 1

try: keep the position of the queue and thread in the class, try GC.Collect () when you get an exception in memory and a call function again. look for the flow to the last position and continue. or: use one database, such as sqlite, and save the latest 100,000 records in each table.