PHP API supports multiple calls

Question

I am making calls to the TransactionSearch PayPal API via PHP cURL.

Unfortunately, the API reacts very slowly, sometimes it takes from 30 seconds to more than 5 minutes (depending on the number of records returned by the API) for one client.

Currently, the script starts the cron job and cycles through each client one at a time. However, if the number of customers increases, the whole process will take a very long time (several hours) to complete. This is not good.

Essentially, I need to run (and process) several API calls at the same time. What is the best way to achieve this?

Answer 1

Since the bottleneck is the remote server, I suggest using curl_multi_exec . You will process a large number of HTTP connections at the same time, and then process their results in a single thread.

This is not the fastest solution that will process responses as soon as they are available in several threads, but this approach can make processing 50 times faster without significant changes.

Answer 2

I would suggest looking at your process in Multi Threading, here is a good question on the stack that has great examples of how to do this.

Answer 3

Using PHP is an easy way to make it multiple processes:

• Create a curlWorker.php file: This processes the CURL request and tracks the result (for example, in the database). It receives the required parameters through the command line as a json string, which is decoded in a script.

• Create the main cron file myCron.php, which executes the loop and calls curlWorker.php with the parameters specified on the command line as the json line:

  for Data-to-process { $cmd = "usr/bin/php path-to-curlWorker.php '$jsonStringParamters' > /dev/null 2>/dev/null &"; exec($cmd); } // example : /usr/bin/php curlWorker.php '{ "uid" : "abc123" , "amount" : 20.3 }' 

• However, you must control how many concurrent execs you can execute on your server, and specify this variable in the main cron file myCron.php