Symfony2 Doctrine2 memory leak / memory limit exceeded

Question

I actually have a lot of problems with the combination of symfony2 and doctrine2. I have to deal with huge data sets (about 2-3 million letters and readings) and make extra efforts to avoid running out of memory.

I found out 2 main points that the memory "leak" (they actually do not leak, but allocate a lot)

• Entitymanager entity store (I don’t know the real name of this) seems to save all processed entites, and you need to clear this storage rule with

    $ entityManager-> clear () 

• Doctrine QueryCache - it caches all the queries used, and the only configuration I found is that you can decide what type of cache you want to use. I did not find that the global ban does not contain a single meaningful value for each request to disable it. Therefore, usually disable it for each request object using the function

   $ qb = $ repository-> createQueryBuilder ($ a);
   $ query = $ qb-> getQuery ();
   $ query-> useQueryCache (false);
   $ query-> execute ();
  

so .. that’s all I understood now. my questions:

Is there an easy way to reject some objects from Entitymanagerstorage? Is there a way to establish the use of querycache in entitymanager? Can I customize this caching behavior in the symonfony doctrine configuration?

It would be great if someone had some good advice for me .. otherwise it might help some newbies.

soa

Answer 1

A little late, but I think I just found a stream on Google Groups from Benjamin Eberlei that answers your question: as indicated , the Doctrine configuration link defaults to register an SQL connection with kernel.debug, so if you created an instance of AppKernel With debug set to true, SQL commands will be stored in memory for each iteration.

You must either create an instance of AppKernel for false, or set logging to false in your YML configuration, or either manually set SQLLogger to null before using EntityManager

$em->getConnection()->getConfiguration()->setSQLLogger(null); 

Answer 2

Try the command with - no-debug . In debug mode, the profiler stores information about each request in memory.

Answer 3

• Set SQL logger to null

$em->getConnection()->getConfiguration()->setSQLLogger(null);

2. Manual call function gc_collect_cycles() after $em->clear()

$em->clear(); gc_collect_cycles();

Remember to set zend.enable_gc to 1 or manually call gc_enable () before use gc_collect_cycles ()

3. Add the --no-debug option if you run the command from the console.

Answer 4

1. Disable logging and profile in app/config/config.yml

 doctrine: dbal: driver: ... ... logging: false profiling: false 

or in code

 $this->em->getConnection()->getConfiguration()->setSQLLogger(null); 

2. Collect the garbage collector . If you are actively using the CPU, then the garbage collector is waiting, and you can quickly find yourself without memory.

First enable manual garbage collection management. Run gc_enable() anywhere in the code. Then run gc_collect_cycles() to force the garbage collector.

Example

 public function execute(InputInterface $input, OutputInterface $output) { gc_enable(); // I'm initing $this->em in __construct using DependencyInjection $customers = $this->em->getRepository('AppBundle:Customer')->findAll(); $counter = 0; foreach ($customers as $customer) { // process customer - some logic here, $this->em->persist and so on if (++$counter % 100 == 0) { $this->em->flush(); // save unsaved changes $this->em->clear(); // clear doctrine managed entities gc_collect_cycles(); // PHP garbage collect // Note that $this->em->clear() detaches all managed entities, // may be you need some; reinit them here } } // don't forget to flush in the end $this->em->flush(); $this->em->clear(); gc_collect_cycles(); } 

If your table is very large, do not use findAll . Use an iterator - http://doctrine-orm.readthedocs.org/projects/doctrine-orm/en/latest/reference/batch-processing.html#iterating-results

Answer 5

received some "funny" news from the symfony doctrine developers in Berlin themselves - they say that in large parties we should not use orm .. it’s just not efficient to create such things in oop

.. yes .. maybe they are right xD

Answer 6

According to standard Doctrine2 documentation, you need to manually clean or delete objects.

In addition to this, when profiling is enabled (as in the default dev environment). The DoctrineBundle in Symfony2 sets up several registrars that use quite a bit of memory. You can completely disable logging, but this is not required.

An interesting side effect is that registrars affect both ORM ORD and DBAL. One of the logs will result in additional memory usage for any service that uses the default logging service. Disabling all of them would be ideal in teams - since the profiler is not yet used.

Here's what you can do to turn off intensive data records while maintaining profiling in other parts of Symfony2:

 $c = $this->getContainer(); /* * The default dbalLogger is configured to keep "stopwatch" events for every query executed * the only way to disable this, as of Symfony 2.3, Doctrine Bundle 1.2, is to reinistiate the class */ $dbalLoggerClass = $c->getParameter('doctrine.dbal.logger.class'); $dbalLogger = new $dbalLoggerClass($c->get('logger')); $c->set('doctrine.dbal.logger', $dbalLogger); // sometimes you need to configure doctrine to use the newly logger manually, like this $doctrineConfiguration = $c->get('doctrine')->getManager()->getConnection()->getConfiguration(); $doctrineConfiguration->setSQLLogger($dbalLogger); /* * If profiling is enabled, this service will store every query in an array * fortunately, this is configurable with a property "enabled" */ if($c->has('doctrine.dbal.logger.profiling.default')) { $c->get('doctrine.dbal.logger.profiling.default')->enabled = false; } /* * When profiling is enabled, the Monolog bundle configures a DebugHandler that * will store every log messgae in memory. * * As of Monolog 1.6, to remove/disable this logger: we have to pop all the handlers * and then push them back on (in the correct order) */ $handlers = array(); try { while($handler = $logger->popHandler()) { if($handler instanceOf \Symfony\Bridge\Monolog\Handler\DebugHandler) { continue; } array_unshift($handlers, $handler); } } catch(\LogicException $e) { /* * As of Monolog 1.6, there is no way to know if there a handler * available to pop off except for the \LogicException that thrown. */ if($e->getMessage() != 'You tried to pop from an empty handler stack.') { /* * this probably doesn't matter, and will probably break in the future * this is here for the sake of people not knowing what they're doing * so than an unknown exception is not silently discarded. */ // remove at your own risk throw $e; } } // push the handlers back on foreach($handlers as $handler) { $logger->pushHandler($handler); } 

Answer 7

Try disabling any existing Doctrine caching. (If you are not using APC / other as a cache, then memory is used).

Delete request cache

 $qb = $repository->createQueryBuilder($a); $query = $qb->getQuery(); $query->useQueryCache(false); $query->useResultCache(false); $query->execute(); 

There is no way to disable it globally

It is also an alternative to clarity, which can help (from here )

 $connection = $em->getCurrentConnection(); $tables = $connection->getTables(); foreach ( $tables as $table ) { $table->clear(); } 

Answer 8

I just wrote a bunch of tips on using Symfony console commands with Doctrine for batch processing here .