C # System.Buffer.BlockCopy Memory problem?

Question

I have a problem that has been bothering me for several days now. I tried to solve the Google problem, but still could not find any solutions, not even one person with the same problem.

It seems that the C # System.Buffer.BlockCopy method leaves you with some ghosts of memory. I have, for example, this method:

private float[,] readFloatArray2 (byte[] b) { int floatSize = sizeof(float); float[,] v = new float[2, (b.Length / 2) / floatSize]; System.Buffer.BlockCopy(b, 0, v, 0, b.Length); return v; } 

to convert an array of bytes to an array of 2D float. Data is pre-read from the stream. I found the problem as a System.Buffer.BlockCopy method.

If I remove the BlockCopy command, the memory used by the application will be half as much. this means it's not my fault that the byte array is still alive. because without the BlockCopy command, the byte array fits correctly. a float is created anyway (with or without copied information).

I'm not quite sure if this is a problem with the BlockCopy or GC command, because I also tried calling System.GC.Collect (); after BlockCopy and then it works great (I know you shouldn't do this ... that's why I ask for advice here).

I also would not ask, but the problem includes several hundred mega.

In addition to memory issues, the method works just fine. Does anyone know what causes a memory problem?

greetings and thanks in advance oli

ps: I am using .NET4.0 with Visual Studio 2010 PRO and WIN7 ... I don’t know how relevant or not.

Answer 1

I found the problem as a System.Buffer.BlockCopy method. If I remove the BlockCopy command, the memory used by the application will be half as much. this means it's not my fault that the byte array is still alive. Because without the BlockCopy command, the BlockCopy array will die correctly.

I do not agree with this conclusion. I see several stages:

• A byte array exists and is populated with data.
• You allocate a float array, but it is not yet populated with data.
• You populate the float array with data
• The byte array is no longer referenced but not yet compiled
• Received an array of bytes.

In a byte array string, BlockCopy is not affected.

Step 2 reserves and fixes virtual memory. Thus, the size of fixation grows at this stage. But since the contents of the array were never written and consist entirely of 00 bytes, the Windows memory manager does not allocate any physical memory for it. He simply notes that these pages consist entirely of 00 s.

The physical memory for the float array only gets the allocation in step 3. You will get the same effect if you added a loop that initialized each field in the array.

---

Besides the actual problem, I also have some constructive suggestions:

• Reuse buffers. GC is good for small short-lived objects, but very bad for large short-lived objects. This means that you should not use functions that allocate and return large arrays. Instead, take them as a parameter, so the existing array can be reused.

• If you use audio work (apparently), I would not use a solid 2D array. Instead, I would use an array of arrays. Where the internal array represents the samples in one buffer, and the external array represents the buffers. This has two advantages:

  • You can easily write code that works on only one channel.
  • Solid 2D arrays are slowly indexed, so they are often faster.
• Are you sure you want to read all the data right away? I read in pieces of a few kilobytes.

Answer 2

BlockCopy does not have a managed .NET implementation. Internally, it causes an external win api.

 [SecuritySafeCritical] public static extern void BlockCopy(Array src, int srcOffset, Array dst, int dstOffset, int count); 

Answer 3

Buffer.BlockCopy is byte based, not index based. I suggest you use Array.Copy , which basically does the same thing. BlockCopy is only a little faster.

you need to convert byte [] to float [] first. See below to

 static float[] ConvertByteArrayToFloat(byte[] bytes) { if(bytes == null) throw new ArgumentNullException("bytes"); if(bytes.Length % 4 != 0) throw new ArgumentException ("bytes does not represent a sequence of floats"); return Enumerable.Range(0, bytes.Length / 4) .Select(i => BitConverter.ToSingle(bytes, i * 4)) .ToArray(); } 

Answer 4

I'm not quite sure if this is a problem with the BlockCopy or GC command, because I also tried calling System.GC.Collect (); after BlockCopy, and then it works fine (I know you shouldn't do this ... that's why I ask for advice here). I would also not have to worry about whether it was not about the few MB hundrets that we are talking about.

The garbage collection starts when there is a need for more memory for a particular generation or from LOH. As a rule, the Garbage Collection does not start just because there is garbage collection, and, as a rule, this is good (we really do not need anything to officially use gigabytes of in-use memory, which we do not use until the GC can receive it, when we need it).

There are times when calling GC.Collect() makes sense in a real program, and this may be one of them, so if it "works fine", then I would not worry too much about what it is against the best practice for 99.9% of the code. The reason this is a “best practice,” and not a tough rule, is that sometimes we are in the 0.1% case, and what is usually best suited ceases to be best practice.

In addition, if you can predict the maximum size of the arrays in advance (or if this is not the case, only the arrays of the original bytes), then the first CodeInChaos approach may work. It really doesn't hurt to use 10,000,000 bytes to process 32, if at some point you really use that 10,000,000. Reusing that 10,000,000 makes very real savings throughout the process.