Welcome all!
I'm having problems with how to fulfill thousands and thousands of requests for a web service (eBay), I have a limited of 5 million calls per day, so there is no problem.
However, I am trying to figure out how to handle 1,000 to 10,000 requests every minute every 5 minutes.
The main stream: 1) Get a list of items from the database (from 1,000 to 10,000 items) 2) Make a POST API request for each element 3) Receive the returned data, process data, update the database
Obviously, one PHP instance executing this in a loop will be impossible.
I know that PHP is not a multi-threaded language.
I tried the CURL solution, mainly: 1) Get a list of elements from the database 2) Initialize a multiple curl session 3) For each element add a curl session for the query 4) perform a multiple curl session
So you can submit 1000-10,000 GET requests ...
This was normal, with about 100-200 requests that occur in about a minute or two, but only 100-200 of the 1000 elements that were processed, I think I find something like an Apache or MySQL limitation?
But this adds latency, which is almost like doing a DoS attack on yourself.
I am wondering how would you deal with this problem? What if you had to make 10,000 requests for web services and 10,000 MySQL updates from the returned data from the web service ... And you need to do this for at least 5 minutes.
PHP MySQL Zend Framework.
!