Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow distinct timeouts for differently labeled handlers in the router #1485

Open
damianr13 opened this issue Aug 25, 2022 · 1 comment · May be fixed by #1560
Open

Allow distinct timeouts for differently labeled handlers in the router #1485

damianr13 opened this issue Aug 25, 2022 · 1 comment · May be fixed by #1560
Assignees
@szmarczak
Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@damianr13
Copy link
Contributor

Describe the feature
I want to be able to specify different timeouts for handlers with different labels. If I expect the website I am crawling to have two or more types of pages, I am doing different things on each of those types, so the time it takes to process a page is different.

Motivation
I am trying to crawl a category page with "infinite scroll" + "load more button" instead of pagination. Similarly to the example in the tutorial (https://crawlee.dev/docs/introduction/scraping), I have 2 types of pages: LIST and DETAIL.

Currently I am facing the problem of hitting a timeout before being able to load all the elements on the LIST page. I looked it up and I found the parameter requestHandlerTimeoutSecs that could be passed to the crawler to increase the timeout limit. My understanding is that this limit applies to all the requests regardless of their types, but I would still like to keep the limit for an individual DETAIL page lower than the high timeout value I need to specify for the LIST page.
@damianr13 damianr13 added the feature Issues that represent new features or improvements to existing features. label Aug 25, 2022
@mnmkng
Copy link
Member

mnmkng commented Aug 25, 2022

Thanks for the feature suggestion @damianr13

@B4nan we should do this. I literally had the same idea 1 hour ago when playing with the router middlewares. I even think we should allow the router to set different preNavigationHooks and other options.

@szmarczak szmarczak self-assigned this Sep 12, 2022
@mstephen19 mstephen19 self-assigned this Sep 21, 2022
@szmarczak szmarczak removed their assignment Sep 21, 2022
@mstephen19 mstephen19 linked a pull request Sep 21, 2022 that will close this issue
Draft
@B4nan B4nan assigned szmarczak and unassigned mstephen19 Oct 10, 2022
@mtrunkat mtrunkat added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@szmarczak szmarczak

Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(Request): add requestHandlerTimeoutSecs and requestTimeoutSecs options apify/crawlee
5 participants
@mtrunkat @damianr13 @mnmkng @szmarczak @mstephen19