I am currently using CURL through a php script that works like a daily cron to export product data in csv format from the site administration area.
The usual way to export data is to go to the "Export" page in the browser and configure the settings, then click the "Export data" button. But since the number of products that I export is very large, and it takes more than 5-10 minutes to export the data, I decided to use the php curl function to simulate this daily via cron.
It used to work fine, but recently, when I increased the number of products in the store by 500+, the script could not return the exported data. By checking it manually, clicking the "Export" button in the browser will return the data correctly. Thus, there is no problem with the timeout when starting the export in the browser manually.
I tested and by removing / decreasing the number of products (thus the required time), the php-curl script works fine again when running cron.
Therefore, I suspect that this is due to a timeout problem, in particular with the curl function in php.
I set CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT to '0' to try. In the php-curl script, I also set "set_time_limit (3000)". But still this does not work, and the request will time out, while the script will not be able to return with the full csv data set.
Any help helping me solve / understand this problem would be greatly appreciated!
Edited: A part of the code has been added where the curl script calls the export function.
$interface = new StoreInterface(); echo "Start exporting at " . date('l jS \of FY h:i:s A') . "\n"; set_time_limit(3000); $result_html = $interface->exportProducts(); //parse $result_html to only retain the csv format preg_match('/<pre>(.*)<\/pre>/s',$result_html[0],$output); if(strlen($output[1])<10) { //debugging for now echo "Export did not happen correctly. Quit\n"; die('Export unsuccessful'); } file_put_contents($output_path,$output[1]); echo "Script completed. Thank you! \n";