Mathematica Big Data Processing

Question

I am dealing with large input arrays with Mathematica, and it looks like I cannot handle anything more than (or equal to) 1024 * 1024 and 81 * 81 * 81. Is this normal? Should I do calculations on such input? If so, how?

Answer 1

I think it depends on what kind of calculations you perform.

For example, in a very modest laptop:

Clear["Global`*"];
k = 2000;
Timing[a = Table[i j + i - j, {i, k}, {j, k}];
 MatrixPlot@a]  

It takes 20 seconds.

Matrix times 1000x1000:

f[n_] := Table[RandomInteger[{1, n}], {n}, {n}];
ListLinePlot[
 Table[{n, First@AbsoluteTiming@(#.#) &@f[n]}, {n, 100, 1000, 100}]]  

So it depends a lot on what you are trying to calculate.