Is there any documentation apart from docs.rs/spider?

[bh4t1a]

Hi! I want to use spider to monitor a website for changes. Is there any documentation which I can read to get started? I tried reading docs.rs/spider but I am unable to figure out how to achieve this.

[j-mendez]

Hi @bh4t1a this example can help get that done https://github.com/spider-rs/spider/blob/main/examples/subscribe.rs. We need to update the intro docs to show the subscribe example since it is a lot more impactful for utilizing the crate fully. In the subscribe example you can get the resource for the asset or page found in the page object with .html.

[bh4t1a]

Thanks! I tried running subscribe.rs example in my local. I get the following error:

Running "target\debug\rs_web_crawler.exe" thread 'main' panicked at 'called "Result::unwrap()" on an "Err" value: Timeout', C:\Users\user1\.cargo\registry\src\index.crates.io-6f17d22bba15001f\spider-1.46.2\src\website.rs:1054:66 note: run with "RUST_BACKTRACE=1" environment variable to display a backtrace error: process didn't exit successfully: "target\debug\rs_web_crawler.exe" (exit code: 101)

I assume it is unable to create a new page using chrome browser. Any idea what might be causing it?

For reference, my Cargo.toml looks like this:

With RUST_BACKTRACE environment variable set to full:
stack_backtrace.txt

[j-mendez]

@bh4t1a it looks like chrome is having trouble launching on the system. L 1054:

    let new_page = browser.new_page("about:blank").await.unwrap();

I have not tested the chrome feature on windows. It may need some changes for improving the launch handling. Here is the repo of the fork of chromiumoxide https://github.com/j-mendez/chromiumoxide. Theres a PR up with some changes to the main repo, that I have not had the time to finish.

In the meantime you can use the CHROME_URL to connect remotely.

[bh4t1a]

Thanks, will do.