Not enough memory with matrix transposition only

Question

I have a cell, Data , it contains three double arrays,

  Data = [74003x253 double] [8061x253 double] [7241x253 double] 

I use a loop to read these arrays and perform some functions,

 for ii = 1 : 3 D = Data {ii} ; m = mean (D') ; // rest of the code end 

Which receives a warning for mean and says:

consider using another DIMENSION input argument for MEAN

However, when I change it,

 for ii = 1 : 3 D = Data {ii}' ; m = mean (D) ; // rest of the code end 

I get an error from memory.

Comparing the two codes, can someone explain what is happening?

It seems that I get an error only with complex conjugate transposition (my data is real).

Answer 1

To take the average value for the nth dimension, you can use mean(D,n) , as already mentioned. Regarding memory consumption, I checked some tests using the Windows Resource Manager. The result was expected.

The D=Data{ii} operation consumes only minimal memory, since here matlab does nothing more than copy the pointer. However, when transposing Matlab, you need to allocate more memory to store matrix D, which means that memory consumption is increasing.

However, this does not exclusively cause memory overflow, since transposition is performed in both cases.

Case 1

Separately in D = Data{ii}';

Case 2

in D = Data {ii}; m = mean(D'); D = Data {ii}; m = mean(D');

The difference is that in case 2, matlab creates a temporary copy of Data{ii}' , which is not saved in the workspace. The allocated memory is the same in both cases, but in case 1, Data{ii}' is stored in D When the memory will later increase, this can lead to memory overflow.

D memory consumption is not so bad (<200 MB), but it is assumed that the memory is already high, and this was enough to ensure memory overflow.

Answer 2

A warning message means that instead

 m = mean (D') ; 

you should:

 m = mean (D,2); 

This will take the average of the second size, leaving you with a column vector of length size(D,1) .

I do not know why you only get an error from memory when you do D = Data {ii}' . Perhaps this is due to the fact that you have it in the direction of mean ( m = mean (D') ; JIT manages to somehow optimize and save your lost memory.

Answer 3

Here are some ways to do this:

 for i = 1 : length(Data) % as chappjc recommends this is an excellent solution m = mean(Data{i}, 2); end 

Or, if you want to transpose, and know that the data is real (not complicated)

 for i = 1 : length(Data) m = mean(Data{i}.'); end 

Note that the dot is before transposition.

Or, skip the loop all together

 m = cellfun(@(d) mean(d, 2), Data, 'uniformoutput', false); 

When you do:

 D = Data{i}' 

Matlab will create a new copy of your data. This will allow you to allocate 74003x253 doubling, which is about 150 MB. As Patrick noted, given that you may have other data, you can easily exceed the allowed memory usage (especially on a 32-bit machine).

If you are working with memory problems, the calculations are insensitive, you can use single precision instead of double, i.e.

 data{i} = single(data{i}); 

Ideally, you want to make the same precision at the selection point to avoid unnecessary new placement and copies.

Good luck.