ZIP - One large file in many calls using PHP PCL Zip

Question

100 MB file → 10 ZIP calls (10 MB per click) → 1 ZIP file

I have to initiate 10 calls to add one 100 MB file to the Zip file (say 10 MB for each call).

The problem is that we have a system with memory and a time limit (which will not process more than 10-15 MB for a call).

So the main idea is to pin a large file with many calls.

I am ready to provide more data if necessary.

Answer 1

Have you ever tried PECL Zip before?

Just zipped two files with the following code without any memory limitation problems. The time limit may be reset. My environment: memory_limit of 3MB and max_ runtime of 20 seconds.

<?php set_time_limit(0); $zip = new ZipArchive(); $zip->open('./test.zip', ZipArchive::CREATE); $zip->addFile('./testa'); // 1.3 GB $zip->addFile('./testb'); // 700mb $zip->close(); 

Note: set_time_limit () will not work on php <5.4 with save_mode = on

---

Another approach might be to create a zip in the background process. This avoids possible memory problems.

Here is an example: http://pastebin.com/7jBaPedb

Using:

 try { $t = new zip('/bin', '/tmp/test.zip'); $t->zip(); if ($t->waitForFinish(100)) echo "Succes :)"; else echo $t->getOutput(); } catch ($e) { echo $e->getMessage(); } 

Instead of waiting for the process to complete, you can write pid in the database and transfer the file if it is finished ...

Answer 2

After reading your question, I first started creating a batch mail packer to do what you requested. It will generate an array with links to a web page that you had to open in order to create a zip file. While the idea worked, I quickly realized that this was not real.

Memory limitation is only a problem when the packer tries to immediately open the entire file and then freeze it. Fortunately, several smart people have already realized that they are easier to make in pieces.

Asbjorn Grandt is one of the people who created this zip class , which is very easy to use and does what you need.

At first I created a very large file. It will be 500 MB in size with various letters in it. This file is a way of reverse processing immediately, which leads to errors with a fatal memory error.

 <?php $fh = fopen("largefile.txt", 'w'); fclose($fh); $fh = fopen("largefile.txt", 'ab'); $size = 500; while($size--) { $l = chr(rand(97, 122)); fwrite($fh, str_repeat($l, 1024*1024)); } fclose($fh); ?> 

And to use the zip class, we will do:

 <?php include('zip.php'); $zip = new Zip(); $zip->setZipFile("largefile.zip"); //the firstname is the name as it will appear inside the zip and the second is the filelocation. In my case I used the same, but you could rename the file inside the zip easily. $zip->addLargeFile("largefile.txt", "largefile.txt"); $zip->finalize(); ?> 

Now a large zip is created in just a few seconds on my server, and the result is a 550 KB file.

Now, if for some strange reason you still need to do this in a few web requests, let me know. I still have the code that I started to do with this.