How to crawl same URL multiple times with different inputs?

[halafi]

I am trying to do in a for loop:

        for (file in files) {
            // I need to pass an info from cycle to the crawler requestHandler
            const crawler = new PlaywrightCrawler({...});
            await crawler.run(["https://foo.bar"]);
            await crawler.teardown();
         }

however this results only the first time the URL get's crawled, so the subsequent runs output:

INFO  PlaywrightCrawler: Starting the crawl
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.

I am probably either doing something very wrong, or miss some bit, any help would be appreciated.

[B4nan]

What version are you on? Probably not on latest.

[halafi]

3.3.2, so should be latest, node v18.16.0

[halafi]

seems for the moment I have to do this workaround:

        for (file in files) {
            // I need to pass an info from about the file to the crawler requestHandler
            const crawler = new PlaywrightCrawler({...});
            if (fileIndex > 0) {
              await crawler.run(["https://foo.bar"]);
            }
            await crawler.run(["https://foo.bar"]);
            await crawler.teardown();
         }

I guess no need for the teardown, but the 2nd call seems to trigger it after the 1st file

[B4nan]

Can you provide complete reproduction? This should work fine since v3.2.2. Note that your code like this is executing the same file twice (you are missing a continue statment there), I doubt you want that.