Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Priority Proxies #35

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

iamashks
Copy link

@iamashks iamashks commented Jan 27, 2020

Hi,

You created a wonderful middleware for Scrapy. It's very useful.

I added a small feature: support for priority proxies. Using this feature, one can add two sets of proxies and give weights to them to use them together. For example, a 0.5 weight says 50% chance of using proxies and 50% chance of using priority_proxies.

I find it useful when I need to use free and premium proxies OR proxies and Tor together. I hope other users may find it useful as well. So, I'm creating a PR.

Kindly review my pull request and provide feedback.

Thank you,
Ash

…of proxies and rotate between them per their weights
…on of proxies and rotate between them per their weights
…ion of proxies and rotate between them per their weights
@iamashks iamashks requested a review from kmike January 27, 2020 12:56
@codecov-io
Copy link

codecov-io commented Jan 27, 2020

Codecov Report

Merging #35 into master will decrease coverage by 5.83%.
The diff coverage is 9.61%.

@@            Coverage Diff             @@
##           master      #35      +/-   ##
==========================================
- Coverage   51.68%   45.84%   -5.84%     
==========================================
  Files           5        5              
  Lines         238      277      +39     
  Branches       39       47       +8     
==========================================
+ Hits          123      127       +4     
- Misses        109      144      +35     
  Partials        6        6
Impacted Files Coverage Δ
rotating_proxies/expire.py 79.34% <100%> (+0.45%) ⬆️
rotating_proxies/middlewares.py 20.6% <6%> (-4.4%) ⬇️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants