Long vector not supported, but error in R Windows 64-bit version

Question

I am trying to check what are the memory limits in the current version of R.

runtest <- function(size) { x <- "testme" while(0<1) { x <- c(x, x) size <<- object.size(x) # size of x when fail } } 

By running runtest(size) in the console on my laptop, I get the following error:

 > runtest(size) Error: cannot allocate vector of size 4.0 Gb In addition: Warning messages: 1: In structure(.Call(C_objectSize, x), class = "object_size") : Reached total allocation of 7915Mb: see help(memory.size) 2: In structure(.Call(C_objectSize, x), class = "object_size") : Reached total allocation of 7915Mb: see help(memory.size) 3: In structure(.Call(C_objectSize, x), class = "object_size") : Reached total allocation of 7915Mb: see help(memory.size) 4: In structure(.Call(C_objectSize, x), class = "object_size") : Reached total allocation of 7915Mb: see help(memory.size) > size 2147483736 bytes > 

This size looks very close to the 2 ^ 31-1 limit that people mentioned earlier. So, I tried to run the same code on our updated desktop with 128 GB of RAM and set a variable in the shortcut for the 64-bit version with a maximum memory capacity of 100 GB. This is the new error I get:

 Error in structure(.Call(C_objectSize, ), class = "object_size"): long vectors not supported yet: unique.c: 1720 > size 8589934680 bytes > 

Does this 8.5 GB limit have anything to do with running on Windows O / S (specifically Windows 7 Enterprise Edition)? I think the R help file ( http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html ) explains this, but it's hard for me to understand what it says ( not my area of ​​expertise)

Answer 1

Looking at the source of size.c and unique.c , it looks like a hash used to improve object.size until it supports long vectors:

 /* Use hashing to improve object.size. Here we want equal CHARSXPs, not equal contents. */ 

and

 /* Currently the hash table is implemented as a (signed) integer array. So there are two 31-bit restrictions, the length of the array and the values. The values are initially NIL (-1). O-based indices are inserted by isDuplicated, and invalidated by setting to NA_INTEGER. */ 

So he object.size choking. How about calling numeric(2^36) to see if it is possible to create such a large object (should be 64 GB).

Answer 2

The maximum vector length in recent versions of R is 2 ^ 53-1, because there are 53 bits in the mantissa in dual memory mode. The maximum size of each dimension of a matrix or array is still 2 ^ 32-1, since the dimension values ​​are still based on integer storage mode. I thought I could get more information from news() , but didn't get as much as I thought. There was quite a lot about this on the r-devel mailing list, and I would use the search in the MarkMail archive if I need more.

 ?double ?integer db <- news() str(db) db$Text[grepl("vector", db$Text) & grepl("length", db$Text)] # only slsightly informative db$Text[grepl("vector", db$Text) & grepl("long", db$Text)][3] 

[1] "There is support for vectors longer than 2 ^ 31 - 1. These are \ napplies for raw, logical, integer, double, complex and characteristic \ nvectors, as well as lists. (Elements of character vectors remain \ n limited to 2 ^ 31 - 1 byte.) "