How does the OS deal with python python that has more memory?

Question

I have a python program that is going to have a lot of memory, primarily in the announcer. This dict will be responsible for assigning a unique integer value to a very large set of keys. Since I work with large matrices, I need a correspondence between the keys, which can also be restored (i.e. as soon as the matrix calculations are finished, I need to match the values ​​with the original keys).

I believe that this amount will ultimately surpass the available memory. I am wondering how this will be handled in relation to swap space. Perhaps a better data structure exists for this purpose.

Answer 1

This will simply end in the swap exchange process because the hash table has very randomized memory access patterns.

If you know that the card exceeds the size of physical memory, you can first use the data structure on the disk. This is especially important if you do not need a data structure during the calculation. When a hash table starts swapping, it also creates problems outside the hash table itself.

Answer 2

You need a database if the data exceeds memory. Indexing dictionaries is not designed for good performance when the dictionary has more memory.

Answer 3

- (python).

dict , redis, . , , :)

Answer 4

, dict, C malloc. , malloc . malloc , , , , . Python MemoryError, malloc , . , dict . - .