Granular crawl

[blinjrm]

Spider seems great for broad crawls, but is it possible to do more granular crawls ?

I really like the enqueueLinks API in Crawlee, as it allows to selectively add links to the queue based on regex, css selector, and more.
Can a similar behavior be implemented with Spider ?

2 use cases I'm trying to implement:

• crawl www.some_website.com/fr, but not www.some_website.com/en or any url that does not start with www.some_website.com/fr/*
• on an e-commerce catalog webpage, crawl all product pages but ignore all other links

[j-mendez]

Yes use website.queue along with subscriptions to add links mid crawl.

[blinjrm]

Yes I saw that example, but does it have to be combined with a subscription? How is a subscription different from a normal crawl?

[j-mendez]

Yes I saw that example, but does it have to be combined with a subscription? How is a subscription different from a normal crawl?

Yes, it kinda needs to be combined with a subscription or a thread. The subscriptions let you get access to the page object concurrently so you do not wait till the end of the crawl.

[blinjrm]

The subscriptions let you get access to the page object concurrently so you do not wait till the end of the crawl

Ok things make a lot more sense now, thanks!

[LiamK]

I have also been struggling with this using the Python wrapper. Is is possible to implement the queue example given above with Python? I don't understand rust, nor whether the proposed answer involving the queue might work with Python.

I ended up getting a Page, getting its list of links, filtering the list with a regex, and then getting the individual pages, but that seems convoluted.

Re: the blacklisting, the regex doesn't accept negative look-aheads. From the rust documentation:

The regex syntax supported by this crate is similar to other regex engines, but it lacks several features that are not known how to implement efficiently. This includes, but is not limited to, look-around and backreferences.

What I would like to do is:

1. start with sub-page, e.g. https://domain.com/events/
2. blacklist urls that don't contain '/event/'

Or possibly in a single positive regex:

1. Get 'https://domain.com/event(s)?/

Without that, I would have to know all the other possible urls on the site and blacklist them. That's impractical and error-prone.

It seems more intuitive to say what I do want, than what I don't. But I can imagine use cases for both approaches.

[j-mendez]

@LiamK this is now possible with whitelisting https://github.com/spider-rs/spider/releases/tag/v1.98.0. Release coming now for the python crate.