Traversing multiple URLs in a loop using a puppeteer

Question

I have

urls = ['url','url','url'...] 

this is what i do

 urls.map(async (url)=>{ await page.goto(`${url}`); await page.waitForNavigation({ waitUntil: 'networkidle' }); }) 

This does not seem to wait for the page to load and quickly types all the URLs (I even tried using page.waitFor)

just wanted to know that I was doing something fundamentally wrong or this type of functionality is not recommended / supported

Answer 1

map , forEach , reduce , etc. Do not wait for an asynchronous operation inside them before moving on to the next element of the iterator that they iterate over.

There are several ways to simultaneously traverse each element of the iterator during an asynchronous operation, but the simplest in this case, I think, would be to simply use the usual for statement, which waits for the operation to complete.

 const urls = [...] for (let i = 0; i < urls.length; i++) { const url = urls[i]; await page.goto(`${url}`); await page.waitForNavigation({ waitUntil: 'networkidle' }); } 

This will visit one URL after another as you expect. If you are curious to repeat the iteration using await / async, you can take a look at this answer: fooobar.com/questions/44161 / ...

Answer 2

If you find that you are waiting for your promise indefinitely, the proposed solution is as follows:

 const urls = [...] for (let i = 0; i < urls.length; i++) { const url = urls[i]; const promise = page.waitForNavigation({ waitUntil: 'networkidle' }); await page.goto('${url}'); await promise; } 

As stated in this github release