Filling FosElasticaBundle ends up with php memory, possible memory leak?

Question

I installed FOSElasticaBundle and am working with a cross section of my data.

My problem arises because I have about 14 m rows that I need to use to build the index. I ran the fill command and after about 6 hours yesterday I made a mistake with a 10.8% memory error:

PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 52277 bytes) in /var/www/html/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php on line 111

As you can see, I set the php memory limit to 2G, which should be pretty excessive.

The last line before the error looked like

Populating index/entity, 10.8% (1315300/12186320), 36 objects/s (RAM : current=2045Mo peak=2047Mo)

Both current and peak ticked with each line starting at 30 MB.

My guess is that there is some kind of memory leak? Of course, php memory should not be exhausted by this process. I also tried a command with some additional parameters

app/console fos:elastica:populate --no-debug --no-reset --env=prod

, , .

, , , ? , , : https://github.com/FriendsOfSymfony/FOSElasticaBundle/issues/82. .

-

Answer 1

,

app/console fos:elastica:populate --no-debug --no-reset --env=prod --offset=n

. , ,

https://github.com/FriendsOfSymfony/FOSElasticaBundle/issues/273

php memory_limit 4G (!) 5 , , , , .

, -, (. https://github.com/FriendsOfSymfony/FOSElasticaBundle/issues/457), memory_limit , , .

Answer 2

, , . , . , .

: , . ( Supervisord ), . .

( forks, pthreads ), enqueue/elastica-bundle, , .

Answer 3

--no-debug , , - fingers_crossed, buffer_size:

monolog:
    handlers:
        main:
            type: fingers_crossed
            action_level: critical
            handler: grouped
            excluded_404s:
                - ^
            buffer_size: 30