Malloc vs DLMalloc on large malloc

Question

I haven't encoded after a while, so excuse me in advance. I have this weird problem. I am trying malloc 8GB at a time, and I plan to manage this bunch later with TLSF. That is, I want to avoid mallocing in my entire application at all, just get one big globe at the beginning and free it at the end. Here is the feature. I have always used dlmalloc so far in my programs. Binding him and everything went well. However, now that I try to malloc 8GB right away and link in dlmalloc to use it, I get segmentation fault 11 on OSX when I run it, without dlmalloc everything is going fine. It doesn't matter if I use gcc or clang. The system does not have 8 GB of RAM, although it has 4 GB. Interestingly, the same thing happens on a Windows machine that has 32 GB of RAM and Ubuntu with 16 GB of RAM. Everything works with the malloc system, the distribution passes, and a simple iteration through the allocated memory works, as expected, on all three systems. But, when I refer to dlmalloc, it fails. Tried this with both malloc function calls and dlmalloc.

The distribution itself is not unusual, simple c99.

[...] size_t bytes = 1024LL*1024LL*1024LL*8LL; unsigned long *m = (unsigned long*)malloc(bytes); [...] 

I am confused by several things. Why does the malloc system give me 8 GB of memory, even without a system with 4 GB or RAM, are these virtual pages? Why doesn't dlmalloc do the same? I know that there cannot be a continuity block of 8 GB of RAM to allocate, but why a segmented error then why not zero ptr?

Is there a viable reliable (hopefully neutral for the platform) solution to get this amount of RAM at a time from malloc, even if I'm not sure that the system will have so much RAM?

edit: the program is 64-bit, like OS ', on which I am running.

edit2: So I played with him a little more. It turns out that if I break the distribution into 1 GB pieces, that is, 8 separate mallocs, then it works with dlmalloc. So this is a problem with continuous range allocation, where dlmalloc is probably trying to allocate only if there is a continuous block. This makes my question even more difficult to formulate. Is there any surefire way to get the size of the memory block with or without dlmalloc on all platforms, and not have it if there is no physical memory to the left (maybe in swap if it does not break). It would also be possible for the cross-platform way to determine if malloc is in ram or swap.

Answer 1

I will give you a little perspective, if not a direct answer. When I see that you are trying to allocate 8 GB of continuous RAM, I cringe. Yes, with 64-bit computing and all that is probably "legal", but on a regular machine you will probably encounter many cross-memory cases, 32-bit legacy code that suffocates in 64-bit size, and just problems with usability, getting a chunk of memory large enough to make this work. If you want to try such things, perhaps try malloc one piece, and then if it fails, use smaller pieces. However, this somewhat strikes the goal of the 1st block. Perhaps the OS has some kind of "page size" to which you could bind your malloc size to help performance and the simple ability to get memory in the desired amount.

In game consoles, this approach to memory management is somewhat common - allocate 1 buffer from the OS when loading as much as possible, and then place your own memory manager there to avoid OS overhead and a possible lower allocation code. It also allows for better control of memory fragmentation on systems where virtual memory does not exist. But on these systems, you also know exactly how much RAM you have.

Is there a way to find out if memory is physical or virtual regardless of platform? I don’t think so, but maybe someone can give a good answer to this question, and I will edit this part.

So it’s not a 100% answer, but some random thoughts to help, and my internal perplexity is what you do that wants 8 GB of RAM in one fragment, when it sounds like multiple pieces will work fine. :)