OutOfMemoryException when creating multiple byte arrays

Question

I constantly encounter an OutOfMemoryException inside a method that creates and processes some byte arrays. The code is as follows:

• Create a MemoryStream to get some data (about 60 MB).
• Create an array of bytes (the same size as a MemoryStream, about 60 MB).
• Fill the array with bytes from the memory stream
• Close MemoryStream
• Processing data from an array of bytes
• Leave method

When this method is called as 20-30 times, I get an OutOfMemoryException to the right where the byte array is allocated. But I do not think that this is a problem with system memory. The application’s memory usage is about 500 MB (private working set), and the test computer is 64-bit with 4 GB of RAM.

Is it possible that the memory used by the byte array or MemoryStream will not be released after the method finishes? But then it does not seem that this memory is allocated for the process, since the private working set is only 500 MB.

What can cause an OutOfMemoryException when creating a large byte array (60 MB) in addition to the lack of physical memory?

[Edited to add sample code] Source comes from PdfSharp lib

An exception is byte[] imageBits = new byte[streamLength]; in the line byte[] imageBits = new byte[streamLength]; It really looks like a LOH fragmentation problem.

 /// <summary> /// Reads images that are returned from GDI+ without color palette. /// </summary> /// <param name="components">4 (32bpp RGB), 3 (24bpp RGB, 32bpp ARGB)</param> /// <param name="bits">8</param> /// <param name="hasAlpha">true (ARGB), false (RGB)</param> private void ReadTrueColorMemoryBitmap(int components, int bits, bool hasAlpha) { int pdfVersion = Owner.Version; MemoryStream memory = new MemoryStream(); image.gdiImage.Save(memory, ImageFormat.Bmp); int streamLength = (int)memory.Length; if (streamLength > 0) { byte[] imageBits = new byte[streamLength]; memory.Seek(0, SeekOrigin.Begin); memory.Read(imageBits, 0, streamLength); memory.Close(); int height = image.PixelHeight; int width = image.PixelWidth; if (ReadWord(imageBits, 0) != 0x4d42 || // "BM" ReadDWord(imageBits, 2) != streamLength || ReadDWord(imageBits, 14) != 40 || // sizeof BITMAPINFOHEADER ReadDWord(imageBits, 18) != width || ReadDWord(imageBits, 22) != height) { throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format"); } if (ReadWord(imageBits, 26) != 1 || (!hasAlpha && ReadWord(imageBits, 28) != components * bits || hasAlpha && ReadWord(imageBits, 28) != (components + 1) * bits) || ReadDWord(imageBits, 30) != 0) { throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format #2"); } int nFileOffset = ReadDWord(imageBits, 10); int logicalComponents = components; if (components == 4) logicalComponents = 3; byte[] imageData = new byte[components * width * height]; bool hasMask = false; bool hasAlphaMask = false; byte[] alphaMask = hasAlpha ? new byte[width * height] : null; MonochromeMask mask = hasAlpha ? new MonochromeMask(width, height) : null; int nOffsetRead = 0; if (logicalComponents == 3) { for (int y = 0; y < height; ++y) { int nOffsetWrite = 3 * (height - 1 - y) * width; int nOffsetWriteAlpha = 0; if (hasAlpha) { mask.StartLine(y); nOffsetWriteAlpha = (height - 1 - y) * width; } for (int x = 0; x < width; ++x) { imageData[nOffsetWrite] = imageBits[nFileOffset + nOffsetRead + 2]; imageData[nOffsetWrite + 1] = imageBits[nFileOffset + nOffsetRead + 1]; imageData[nOffsetWrite + 2] = imageBits[nFileOffset + nOffsetRead]; if (hasAlpha) { mask.AddPel(imageBits[nFileOffset + nOffsetRead + 3]); alphaMask[nOffsetWriteAlpha] = imageBits[nFileOffset + nOffsetRead + 3]; if (!hasMask || !hasAlphaMask) { if (imageBits[nFileOffset + nOffsetRead + 3] != 255) { hasMask = true; if (imageBits[nFileOffset + nOffsetRead + 3] != 0) hasAlphaMask = true; } } ++nOffsetWriteAlpha; } nOffsetRead += hasAlpha ? 4 : components; nOffsetWrite += 3; } nOffsetRead = 4 * ((nOffsetRead + 3) / 4); // Align to 32 bit boundary } } else if (components == 1) { // Grayscale throw new NotImplementedException("Image format not supported (grayscales)."); } FlateDecode fd = new FlateDecode(); if (hasMask) { // monochrome mask is either sufficient or // provided for compatibility with older reader versions byte[] maskDataCompressed = fd.Encode(mask.MaskData); PdfDictionary pdfMask = new PdfDictionary(document); pdfMask.Elements.SetName(Keys.Type, "/XObject"); pdfMask.Elements.SetName(Keys.Subtype, "/Image"); Owner.irefTable.Add(pdfMask); pdfMask.Stream = new PdfStream(maskDataCompressed, pdfMask); pdfMask.Elements[Keys.Length] = new PdfInteger(maskDataCompressed.Length); pdfMask.Elements[Keys.Filter] = new PdfName("/FlateDecode"); pdfMask.Elements[Keys.Width] = new PdfInteger(width); pdfMask.Elements[Keys.Height] = new PdfInteger(height); pdfMask.Elements[Keys.BitsPerComponent] = new PdfInteger(1); pdfMask.Elements[Keys.ImageMask] = new PdfBoolean(true); Elements[Keys.Mask] = pdfMask.Reference; } if (hasMask && hasAlphaMask && pdfVersion >= 14) { // The image provides an alpha mask (requires Arcrobat 5.0 or higher) byte[] alphaMaskCompressed = fd.Encode(alphaMask); PdfDictionary smask = new PdfDictionary(document); smask.Elements.SetName(Keys.Type, "/XObject"); smask.Elements.SetName(Keys.Subtype, "/Image"); Owner.irefTable.Add(smask); smask.Stream = new PdfStream(alphaMaskCompressed, smask); smask.Elements[Keys.Length] = new PdfInteger(alphaMaskCompressed.Length); smask.Elements[Keys.Filter] = new PdfName("/FlateDecode"); smask.Elements[Keys.Width] = new PdfInteger(width); smask.Elements[Keys.Height] = new PdfInteger(height); smask.Elements[Keys.BitsPerComponent] = new PdfInteger(8); smask.Elements[Keys.ColorSpace] = new PdfName("/DeviceGray"); Elements[Keys.SMask] = smask.Reference; } byte[] imageDataCompressed = fd.Encode(imageData); Stream = new PdfStream(imageDataCompressed, this); Elements[Keys.Length] = new PdfInteger(imageDataCompressed.Length); Elements[Keys.Filter] = new PdfName("/FlateDecode"); Elements[Keys.Width] = new PdfInteger(width); Elements[Keys.Height] = new PdfInteger(height); Elements[Keys.BitsPerComponent] = new PdfInteger(8); // TODO: CMYK Elements[Keys.ColorSpace] = new PdfName("/DeviceRGB"); if (image.Interpolate) Elements[Keys.Interpolate] = PdfBoolean.True; } } 

Answer 1

Hope you use MemoryStream.GetBuffer() and that you are not copying the new array.

Your main problem is not a direct lack of memory, but LOH fragmentation. This can be a difficult problem, the main problem is the allocation of large buffers of different sizes. Elements on the LOH are assembled but not compacted.

Solutions may be:

• first make sure that you are not blocking anything from collecting. use a profiler.
• try reusing your buffer (s).
• Round distributions up for a set of fixed numbers.

The last 2 require you to work with large arrays, it may take some work.

Answer 2

Disposal has been proposed, but only for MemoryStream it will do nothing. Also suggested by GC.Collect . This may not help, as it seems that your memory is not growing significantly. Be careful, calling GC.Collect can be an expensive operation.

Splitting up

It looks like you are facing the notorious problem of fragmentation of large objects. This can be caused by the fact that often allocate and free up pieces of memory of 60 MB. If LOH is fragmented, it remains fragmented. This is a serious problem with long-running .NET applications and the reason ASP.NET is often configured to restart at intervals.

• fooobar.com/questions/13420 / ...
• Great explanation and easy to follow code on Codeproject .
• Submitted to Microsoft , but nothing will be done.

OutOfMemoryException Prevention

See the CodeProject article above on how to do this. The trick is to use a MemoryFailPoint and catch an InsufficientMemoryException . Thus, you can competently degrade, and your application will not become unstable.

Possible common solution

Make sure your large objects live as long as possible. Reuse buffer. Select it once with sufficient size and zero buffer when you need it. This way you will not encounter other memory problems. When your objects remain below 85 KB, they usually do not fall into LOH and will not interfere.

64-bit machines should not have this problem

EDIT: According to this post (temporary tab) and this post (see open comment), this problem should not appear on 64-bit machines. Since you are saying that you are running your code on 64-bit machines, are you possibly compiled with configuring on x86?

Answer 3

Your heap memory throws this exception, try calling GC.Collect () at the end to free up resources. You can also MemoryProfiler learn how to use heap memory, it comes with a trial version of 14 days

Answer 4

Try enclosing it in a using(MemoryStream x = ...) { } block that will delete the object for you.

Although Close intended to be a Dispose object, as recommended by .NET, it may be different from a MemoryStream .

Answer 5

I have the same problem in the code as shown below:

 ImageData = (byte[])pDataReader.qDataTable.Rows[0][11]; if (ImageData != null) { ms = new MemoryStream(ImageData); button1.BackgroundImage = Image.FromStream(ms); } ImageData = null; ImageData = (byte[])pDataReader.qDataTable.Rows[0][12]; if (ImageData != null) { ms = new MemoryStream(ImageData); button1.BackgroundImage = Image.FromStream(ms); } ImageData = null; ms.Close(); 

Removing ms.Close(); solves the problem.
I think the problem is increasing because you are defining MemoryStream memory = new MemoryStream(); outside the if block and close it in the if block, like me!

Answer 6

First of all, it is not recommended to read or write big FILESTREAM data through TSQL. The recommended approach is to use the Win32 / DOTNET APIs provided by SQL Server. The code I posted above shows how to access the FILESTREAM data using the SqlFileStream (). Net class. It also shows how to send data into smaller pieces.

By providing sufficient shared memory, you can prevent memory exceptions caused by LOH fragmentation by creating a bunch of smaller arrays and wrapping them in one IList or some other indexed interface.