Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve concurrent downloading capacity of script. #18

Open
ellakdev opened this issue Sep 25, 2024 · 3 comments · May be fixed by #26
Open

Improve concurrent downloading capacity of script. #18

ellakdev opened this issue Sep 25, 2024 · 3 comments · May be fixed by #26
Assignees
@nikostsekos
Labels
good first issue Good for newcomers

Comments

@ellakdev
Copy link
Collaborator

ellakdev commented Sep 25, 2024 •
edited by fffoivos
Loading

Improve concurrent downloading capacity of script.
@fffoivos fffoivos changed the title Fix downloader script Improve concurrent downloading capacity of script. Sep 29, 2024
@fffoivos fffoivos added the good first issue Good for newcomers label Sep 29, 2024
@Sadique982
Copy link

Can i work on this issue?

@fffoivos
Copy link
Collaborator

fffoivos commented Oct 25, 2024 •
edited
Loading

Hi @Sadique982,

Sure, look at scraping/download_and_extract_scripts/downloader.py and try to increase the PDFs per minute downloading at scraping/json_sitemaps/pergamos_list_pdf.json . Make sure you don't DDOS the site and you don't get blocked! One option if blocked is trying to implement multiple downloaders with torify.

Sadique982 added a commit to Sadique982/glossAPI that referenced this issue Nov 2, 2024
@Sadique982
[Improve concurrent downloading capacity of script. eellak#18]
e4e5d68 
In this update, I have increased the number of PDFs downloaded per minute while ensuring that we do not overwhelm the server. The changes can be found in `scraping/download_and_extract_scripts/downloader.py`. I’ve implemented methods to manage concurrent downloads more effectively, including utilizing semaphore limits and adding sleep intervals. Additionally, I have considered strategies to avoid getting blocked, such as implementing multiple downloaders with `torify` if necessary.
@Sadique982 Sadique982 mentioned this issue Nov 2, 2024
Closed
@rajku-dev
Copy link

I want to work on this issue.

@rajku-dev rajku-dev linked a pull request Nov 12, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikostsekos nikostsekos

Labels
good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

changes to increase download capacity rajku-dev/glossAPI
5 participants
@ellakdev @fffoivos @Sadique982 @rajku-dev @nikostsekos