This is not about MATLAB. What happens when you try to allocate more memory than exists in your hardware is the behavior of a particular OS.
On Linux, by default, the OS is “optimistic” about almost everything you need, i.e. swap space is also considered allocated memory. You will get what you want - there is no OOM error, but slow calculations with the data highlighted by the subcap. This "function" is called overcommit. You can change this behavior by changing the rebuild settings on Linux (see here for a brief summary).
Overcommit is probably not the best idea to use this to solve big problems in MATLAB, as the whole OS starts to work very slowly. It can definitely not be compared with optimized "extracardiac" implementations that deliberately use the hard drive in computing.
Here's how to do it on Linux. I don't know how to change the memory allocation behavior in Windows, but I doubt that you really want it. You need more RAM.
And you confuse things - exchange has nothing to do with continuous memory. The memory allocated by the OS is “virtual memory” that is contiguous regardless of whether the underlying physical memory can be mapped to adjacent pages.
Change For transparent “non-root” operations with large matrices that use disk space as extra memory, you might want to take a look at the VVA fileexchange project . This class claims to be a regular MATLAB class, but it works with the base file of the hard drive. Note that the usual MATLAB array size limits are usually applied.