Piece cache fixes and improvements #3238
Merged
+315
−53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…atch after partial completion of the previous batches
shamil-gadelshin
approved these changes
Nov 18, 2024
| @@ -647,6 +647,10 @@ where | |||
| if let Some(peers) = pieces_to_download.get_mut(&piece_index) { | |||
| peers.extend(Vec::from(closest_peers)); | |||
| } | |||
|
|
|||
| // No need to ask this peer again if they didn't have the piece we expected, or | |||
| // they claimed to have earlier | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This both fixes and significantly improves piece cache sync and piece downloading more generally.
One bug was not storing free offset in case more than one cache stores the same piece index (usually caused by user manipulation with farms), test case was added to check this.
Significant slowdown was caused by faulty peers that initially claim to have a bunch of pieces, but then not returning them and instead returning closes peers, which in pathological cases meant farmer was sending thousands of individual requests to get pieces to a single peer one at a time, but was only getting closest peers back, meaning it was wasting a lot of time and to user that looked like downloading hanged even though it technically didn't.
I also improved piece cache sync performance by allowing to schedule more batches after previous batches partially completed, which helps with bandwidth saturation, this allowed me to download pieces as 100-250 Mbps speed most of the time, with higher limits can push closer to 800 Mbps. For higher utilization we'll need to introduce CLI option to increase memory usage by piece cache sync (right now constrained to use around 1GB).
Code contributor checklist: