Array - out of memory in php

Question

I have a large array of associative arrays. Each associative array consists of approximately 15 keys of different types (string, integer, float). Example below:

$array = [ [ "key1" => "string", "key2" => 10, "key3" => 4.05 ], [ "key1" => "string2", "key2" => 20, "key3" => 1.05 ], ... ]; 

Now I want to iterate over this array and add some keys like

 $map = array_map(function (array $item) { $item['key4'] = 1; $item['key5'] = 1; $item['key6'] = 1; return $item; }, $array); 

Problem: For an array that contains a fairly large number of associative arrays, adding new keys makes the memory limit reached and the script exits. Do you have any solutions?

Answer 1

You can paginate your data, block your array to work with smaller parts, or even increase memory_limit, but suppose you have a large array and cannot do otherwise.

So, let's play with an array of 1,000,000 and try different solutions. I put a measurement of memory consumption and timing from my laptop.

Current Solution (857MB / 640ms)

 for ($i=0; $i< 1000000; $i++){ $array[$i] = [ "key" => 'value', "key2" => $i, "key3" => $i / 3 ]; } $map = array_map(function (array $item) { $item['key4'] = 1; $item['key5'] = 1; $item['key6'] = 1; return $item; }, $array); 

With this piece of code, the memory consumption on my laptop is 857 MB, and the calculation time is 640 ms .

In your example, you create a whole new $map variable from your $array . This means that you are creating a new copy of the array in memory.

Work with references (480MB / 220ms)

 $array = []; for ($i=0; $i< 1000000; $i++){ $array[$i] = [ "key" => 'value', "key2" => $i, "key3" => $i / 3 ]; } foreach ($array as &$item) { $item['key4'] = 1; $item['key5'] = 1; $item['key6'] = 1; } 

Using &$item we ask PHP to give us access to the variable by reference, which means that we modify the data directly in memory without creating a new copy.

This is why this script consumes much less memory and calculates time.

Work with classes (223MB / 95ms)

Under the hood, PHP uses C data structures to manage in-memory data. Classes are predictable and it is much easier to optimize PHP than an array. This is well explained here.

 class TestClass { public $key1, $key2, $key3, $key4, $key5, $key6; } $array = []; for ($i=0; $i< 1000000; $i++){ $array[$i] = new TestClass(); $array[$i]->key1 = 'value'; $array[$i]->key2 = $i; $array[$i]->key3 = $i / 3; } foreach ($array as $item) { $item->key4 = 1; $item->key5 = 1; $item->key6 = 1; } 

You can see that memory consumption and iteration time are much lower. This is due to the fact that PHP does not need to change the structure of the data in memory: each field of the object is ready to receive data.

Be careful if you add a field that was not declared in the class (for example, $item->newKey = 1 : newKey is declared dynamically): memory optimization will no longer be possible, and you will go to memory 620 MB of use and 280 ms)

---

If you want to go further and not be afraid of headaches, take a look at the PHP Standard Library (SPL): you will find many optimized data structure solutions (Fixed Array, Iterators, etc.)

PS: test with Xdebug disabled

Answer 2

You can save some memory if you use links. This will remove many copies of the recording actions that occur in the background. In a small case. I was able to reduce memory by 30% -40% (depending on the version of PHP). If you use PHP5, you can also profit from upgrading to PHP7. Obviously, I cannot predict if one or both retains sufficient memory. Test case (just delete /* in front of the map or take a walk):

 $cnt=10000; for($i=0;$i<$cnt;$i++) { $array[]['key1'] = 1; $array[]['key2'] = 2; $array[]['key3'] = 3; } /*array_walk($array, function (&$item,$key) { $item['key4'] = 1; $item['key5'] = 1; $item['key6'] = 1; }); //memory used PHP7/PHP5: 13 437 720 - 25 924 944 */ /*$map = array_map(function (array $item) { $item['key4'] = 1; $item['key5'] = 1; $item['key6'] = 1; return $item; }, $array); //memory used PHP7/PHP5: 25 050 360 - 40 850 480*/