why Express duplicate request Crawler not running?

[jeff-fe]

I use express and crawlee create nodejs serve, repeated query to an interface, Crawler only run once.
I search issue find Results. add uniqueKey can solve this question.

So I would like to ask a question？ why need add uniqueKey? thanks very much

"nodejs" : 16.19.1
"crawlee": "^3.4.0",
"express": "^4.18.2"

import express from "express";
import { BasicCrawler } from 'crawlee';

const app = express()
const port = 8001

app.get('/', async (req, res) => {
  const urlArray = []
  const crawler = new BasicCrawler({
    async requestHandler({ request }) {
      const { url } = request;
      urlArray.push(url)
    },
  });

  await crawler.run([
    {
      url: 'http://www.baidu.com',
      // uniqueKey: new Date().getTime() + ''
    }
  ]);

  res.send({
    data: urlArray,
    timeStamp: new Date().toLocaleString(),
  })
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

[B4nan]

Are you really on v3.4? Since v3.3.2 the run call resets the default RQ state automatically.

[jeff-fe]

I'm sure, crawlee version 3.4.0. you can run top express demo. repeat query the inrterface, crawle run once. second query not run.

[B4nan]

The problem in your repro is that you are not using the same crawler instance, so the call to run method in fact does not happen twice (not on the same crawler instance, which is what triggers the auto clearing of RQ).

[B4nan]

This was originally a fail-safe, as you might prepare the requests in the queue upfront, and you don't want to wipe those. Maybe we could allow wiping the RQ via the purgeRequestQueue option of the run method, even for the first run if you explicitly provide true in there.

[jeff-fe]

ok thank you very much!

[richardzen]

@jeff-fe I am facing the same issue, crawlee 3.9.2. Can you please tell me how did you fix it? Thank you