Memory usage runs counter to Doctrine bulk board

Question

I am trying to insert a large amount of data (30,000+ rows) into a MySQL database using Doctrine2 and the Symfony2 toolkit.

I looked at the right way to do this . I saw many questions about memory leaks and the Doctrine, but there was no satisfactory answer for me. The Doctrine function is often used clear().

So, I made various forms of this:

while (($data = getData()) {
    $iteration++;

    $obj = new EntityObject();
    $obj->setName('henry');
    // Fill object...

    $manager->persist($obj);

    if ($iteration % 500 == 0) {
        $manager->flush();
        $manager->clear();

        // Also tried some sort of:
        // $manager->clear($obj);   
        // $manager->detach($obj);
        // gc_collect_cycles();
    }
}

PHP - , flush() ( ). , , , , . , : , KB.

clear(), detach() GC . KB.

? - ? ?

:

• flush() ;
• ;
• CSV, ;

EDIT ( ):

@qooplmao , , sql logger: $manager->getConnection()->getConfiguration()->setSQLLogger(null);

, - .

Answer 1

, , @Axalix.

:

// IMPORTANT - Disable the Doctrine SQL Logger
$manager->getConnection()->getConfiguration()->setSQLLogger(null);

// SUGGESION - make getData as a generator (using yield) to to save more memory.
while ($data = getData()) {
  $iteration++;

  $obj = new EntityObject();
  $obj->setName('henry');
  // Fill object...

  $manager->persist($obj);

  // IMPORTANT - Temporary store entities (of course, must be defined first outside of the loop)
  $tempObjets[] = $obj;

  if ($iteration % 500 == 0) {
    $manager->flush();

    // IMPORTANT - clean entities
    foreach($tempObjets as $tempObject) {
      $manager->detach($tempObject);
    }

    $tempObjets = null;
    gc_enable();
    gc_collect_cycles();
  }
}

// Do not forget the last flush
$manager->flush();

, : script Symfony, --no-debug . .

Answer 2

- Doctrine . , .

MySQL , LOAD DATA. csv LOAD .

, csv $csvData = array_map("str_getcsv", file($csv));. , , . .csv LOAD MySQL.

, Doctrine , .