Parsing huge XML files in PHP

Question

I am trying to parse XML content files / DMOZ structures in MySQL, but all existing scripts for this are very old and do not work well. How can I open a large (+ 1 GB) XML file in PHP for parsing?

Answer 1

There are only two php APIs that are really suitable for handling large files. Firstly, the old emigrant api, and the second is the newer XMLReader functions. These APIs read continuous streams rather than loading the entire tree into memory (which Simplexml and DOM do).

For example, you can look at this partial DMOZ directory parser:

<?php class SimpleDMOZParser { protected $_stack = array(); protected $_file = ""; protected $_parser = null; protected $_currentId = ""; protected $_current = ""; public function __construct($file) { $this->_file = $file; $this->_parser = xml_parser_create("UTF-8"); xml_set_object($this->_parser, $this); xml_set_element_handler($this->_parser, "startTag", "endTag"); } public function startTag($parser, $name, $attribs) { array_push($this->_stack, $this->_current); if ($name == "TOPIC" && count($attribs)) { $this->_currentId = $attribs["R:ID"]; } if ($name == "LINK" && strpos($this->_currentId, "Top/Home/Consumer_Information/Electronics/") === 0) { echo $attribs["R:RESOURCE"] . "\n"; } $this->_current = $name; } public function endTag($parser, $name) { $this->_current = array_pop($this->_stack); } public function parse() { $fh = fopen($this->_file, "r"); if (!$fh) { die("Epic fail!\n"); } while (!feof($fh)) { $data = fread($fh, 4096); xml_parse($this->_parser, $data, feof($fh)); } } } $parser = new SimpleDMOZParser("content.rdf.u8"); $parser->parse(); 

Answer 2

This is a very similar question to the Best way to handle large XML in PHP , but with a very good concrete answer aimed at solving the specific problem of parsing the DMOZ directory. However, since this is a good Google hit for large XML in general, I will also post my answer from another question:

My occupation:

https://github.com/prewk/XmlStreamer

A simple class that will extract all children to the root XML element when streaming a file. Tested on 108 MB XML file with pubmed.com.

 class SimpleXmlStreamer extends XmlStreamer { public function processNode($xmlString, $elementName, $nodeIndex) { $xml = simplexml_load_string($xmlString); // Do something with your SimpleXML object return true; } } $streamer = new SimpleXmlStreamer("myLargeXmlFile.xml"); $streamer->parse(); 

Answer 3

I recently had to parse some fairly large XML documents, and I need a method to read one element at a time.

If you have the following complex-test.xml :

 <?xml version="1.0" encoding="UTF-8"?> <Complex> <Object> <Title>Title 1</Title> <Name>It name goes here</Name> <ObjectData> <Info1></Info1> <Info2></Info2> <Info3></Info3> <Info4></Info4> </ObjectData> <Date></Date> </Object> <Object></Object> <Object> <AnotherObject></AnotherObject> <Data></Data> </Object> <Object></Object> <Object></Object> </Complex> 

And wanted to return <Object/> s

PHP:

 require_once('class.chunk.php'); $file = new Chunk('complex-test.xml', array('element' => 'Object')); while ($xml = $file->read()) { $obj = simplexml_load_string($xml); // do some parsing, insert to DB whatever } ########### Class File ########### <?php /** * Chunk * * Reads a large file in as chunks for easier parsing. * * The chunks returned are whole <$this->options['element']/>s found within file. * * Each call to read() returns the whole element including start and end tags. * * Tested with a 1.8MB file, extracted 500 elements in 0.11s * (with no work done, just extracting the elements) * * Usage: * <code> * // initialize the object * $file = new Chunk('chunk-test.xml', array('element' => 'Chunk')); * * // loop through the file until all lines are read * while ($xml = $file->read()) { * // do whatever you want with the string * $o = simplexml_load_string($xml); * } * </code> * * @package default * @author Dom Hastings */ class Chunk { /** * options * * @var array Contains all major options * @access public */ public $options = array( 'path' => './', // string The path to check for $file in 'element' => '', // string The XML element to return 'chunkSize' => 512 // integer The amount of bytes to retrieve in each chunk ); /** * file * * @var string The filename being read * @access public */ public $file = ''; /** * pointer * * @var integer The current position the file is being read from * @access public */ public $pointer = 0; /** * handle * * @var resource The fopen() resource * @access private */ private $handle = null; /** * reading * * @var boolean Whether the script is currently reading the file * @access private */ private $reading = false; /** * readBuffer * * @var string Used to make sure start tags aren't missed * @access private */ private $readBuffer = ''; /** * __construct * * Builds the Chunk object * * @param string $file The filename to work with * @param array $options The options with which to parse the file * @author Dom Hastings * @access public */ public function __construct($file, $options = array()) { // merge the options together $this->options = array_merge($this->options, (is_array($options) ? $options : array())); // check that the path ends with a / if (substr($this->options['path'], -1) != '/') { $this->options['path'] .= '/'; } // normalize the filename $file = basename($file); // make sure chunkSize is an int $this->options['chunkSize'] = intval($this->options['chunkSize']); // check it valid if ($this->options['chunkSize'] < 64) { $this->options['chunkSize'] = 512; } // set the filename $this->file = realpath($this->options['path'].$file); // check the file exists if (!file_exists($this->file)) { throw new Exception('Cannot load file: '.$this->file); } // open the file $this->handle = fopen($this->file, 'r'); // check the file opened successfully if (!$this->handle) { throw new Exception('Error opening file for reading'); } } /** * __destruct * * Cleans up * * @return void * @author Dom Hastings * @access public */ public function __destruct() { // close the file resource fclose($this->handle); } /** * read * * Reads the first available occurence of the XML element $this->options['element'] * * @return string The XML string from $this->file * @author Dom Hastings * @access public */ public function read() { // check we have an element specified if (!empty($this->options['element'])) { // trim it $element = trim($this->options['element']); } else { $element = ''; } // initialize the buffer $buffer = false; // if the element is empty if (empty($element)) { // let the script know we're reading $this->reading = true; // read in the whole doc, cos we don't know what wanted while ($this->reading) { $buffer .= fread($this->handle, $this->options['chunkSize']); $this->reading = (!feof($this->handle)); } // return it all return $buffer; // we must be looking for a specific element } else { // set up the strings to find $open = '<'.$element.'>'; $close = '</'.$element.'>'; // let the script know we're reading $this->reading = true; // reset the global buffer $this->readBuffer = ''; // this is used to ensure all data is read, and to make sure we don't send the start data again by mistake $store = false; // seek to the position we need in the file fseek($this->handle, $this->pointer); // start reading while ($this->reading && !feof($this->handle)) { // store the chunk in a temporary variable $tmp = fread($this->handle, $this->options['chunkSize']); // update the global buffer $this->readBuffer .= $tmp; // check for the open string $checkOpen = strpos($tmp, $open); // if it wasn't in the new buffer if (!$checkOpen && !($store)) { // check the full buffer (in case it was only half in this buffer) $checkOpen = strpos($this->readBuffer, $open); // if it was in there if ($checkOpen) { // set it to the remainder $checkOpen = $checkOpen % $this->options['chunkSize']; } } // check for the close string $checkClose = strpos($tmp, $close); // if it wasn't in the new buffer if (!$checkClose && ($store)) { // check the full buffer (in case it was only half in this buffer) $checkClose = strpos($this->readBuffer, $close); // if it was in there if ($checkClose) { // set it to the remainder plus the length of the close string itself $checkClose = ($checkClose + strlen($close)) % $this->options['chunkSize']; } // if it was } elseif ($checkClose) { // add the length of the close string itself $checkClose += strlen($close); } // if we've found the opening string and we're not already reading another element if ($checkOpen !== false && !($store)) { // if we're found the end element too if ($checkClose !== false) { // append the string only between the start and end element $buffer .= substr($tmp, $checkOpen, ($checkClose - $checkOpen)); // update the pointer $this->pointer += $checkClose; // let the script know we're done $this->reading = false; } else { // append the data we know to be part of this element $buffer .= substr($tmp, $checkOpen); // update the pointer $this->pointer += $this->options['chunkSize']; // let the script know we're gonna be storing all the data until we find the close element $store = true; } // if we've found the closing element } elseif ($checkClose !== false) { // update the buffer with the data upto and including the close tag $buffer .= substr($tmp, 0, $checkClose); // update the pointer $this->pointer += $checkClose; // let the script know we're done $this->reading = false; // if we've found the closing element, but half in the previous chunk } elseif ($store) { // update the buffer $buffer .= $tmp; // and the pointer $this->pointer += $this->options['chunkSize']; } } } // return the element (or the whole file if we're not looking for elements) return $buffer; } } 

Answer 4

I would suggest using a SAX-based parser rather than a DOM-based parsing.

Information on using SAX in PHP: http://www.brainbell.com/tutorials/php/Parsing_XML_With_SAX.htm

Answer 5

This is not a great solution, but just need to add another option:

You can break many large XML files into chunks, especially those that are actually lists of similar elements (since I suspect that the file you are working with will be).

for example, if your document looks like this:

 <dmoz> <listing>....</listing> <listing>....</listing> <listing>....</listing> <listing>....</listing> <listing>....</listing> <listing>....</listing> ... </dmoz> 

You can read it in mega or two at a time, artificially wrap several full <listing> tags loaded into the root level tag, and then load them through simplexml / domxml (I used domxml when using this approach).

Frankly, I prefer this approach if you use PHP <5.1.2. XMLReader is available with 5.1.2 and higher, which is probably the best option, but before that you were stuck with either the aforementioned chunking strategy or the old SAX / expat lib. And I don’t know about all of you, but I HATE writing / maintaining SAX / expat parsers.

Please note, however, that this approach is NOT really practical when your document does not consist of many identical lower-level elements (for example, it works great for any list of files or URLs, etc., but it won’t make sense to parse a large HTML document)

Answer 6

This is an old post, but first in Google search results, so I decided to post another solution based on this post:

http://drib.tech/programming/parse-large-xml-files-php

This solution uses both XMLReader and SimpleXMLElement :

 $xmlFile = 'the_LARGE_xml_file_to_load.xml' $primEL = 'the_name_of_your_element'; $xml = new XMLReader(); $xml->open($xmlFile); // finding first primary element to work with while($xml->read() && $xml->name != $primEL){;} // looping through elements while($xml->name == $primEL) { // loading element data into simpleXML object $element = new SimpleXMLElement($xml->readOuterXML()); // DO STUFF // moving pointer $xml->next($primEL); // clearing current element unset($element); } // end while $xml->close(); 

Answer 7

You can combine XMLReader with DOM for this. In PHP, both APIs (and SimpleXML) are based on the same library - libxml2. Large XML files are usually a list of records. This way you use XMLReader to iterate over the records, load one record into the DOM, and use the DOM and Xpath methods to retrieve the values. The key is the XMLReader::expand() method. It loads the current node into an instance of XMLReader and its descendants as DOM nodes.

XML example:

 <books> <book> <title isbn="978-0596100087">XSLT 1.0 Pocket Reference</title> </book> <book> <title isbn="978-0596100506">XML Pocket Reference</title> </book> <!-- ... --> </books> 

Code example:

 // open the XML file $reader = new XMLReader(); $reader->open('books.xml'); // prepare a DOM document $document = new DOMDocument(); $xpath = new DOMXpath($document); // find the first 'book' element node at any depth while ($reader->read() && $reader->localName !== 'book') { continue; } // as long as here is a node with the name "book" while ($reader->localName === 'book') { // expand the node into the prepared DOM $book = $reader->expand($document); // use Xpath expressions to fetch values var_dump( $xpath->evaluate('string(title/@isbn)', $book), $xpath->evaluate('string(title)', $book) ); // move to the next book sibling node $reader->next('book'); } $reader->close(); 

Note that the extended node is never added to the DOM document. This allows the GC to clear it.

This approach also works with XML namespaces.

 $namespaceURI = 'urn:example-books'; $reader = new XMLReader(); $reader->open('books.xml'); $document = new DOMDocument(); $xpath = new DOMXpath($document); // register a prefix for the Xpath expressions $xpath->registerNamespace('b', $namespaceURI); // compare local node name and namespace URI while ( $reader->read() && ( $reader->localName !== 'book' || $reader->namespaceURI !== $namespaceURI ) ) { continue; } // iterate the book elements while ($reader->localName === 'book') { // validate that they are in the namespace if ($reader->namespaceURI === $namespaceURI) { $book = $reader->expand($document); var_dump( $xpath->evaluate('string(b:title/@isbn)', $book), $xpath->evaluate('string(b:title)', $book) ); } $reader->next('book'); } $reader->close();