beam_search fix for running with torch.use_deterministic_algorithms(True) #1096
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rue) Context: Pytorch has some bugs related to setting values for tensors to scalars when using torch.use_deterministic_algorithms that were only addressed in most recent Pytorch versions. See: * pytorch/pytorch@2d4b1ae * pytorch/pytorch#68525 The scalar attribution in beam_search.forward would throw: ``` RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/cuda/Indexing.cu":250, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor31 ``` when running with torch.use_deterministic_algorithms(True). The change in the CR did not have any changes in performance, but works around the 2 bugs mentioned above. Testing done: * Ran with a custom model, using `torch.use_deterministic_algorithms(True)`
Jehovan
requested review from
fhieber,
tdomhan,
mjdenkowski,
xingniu and
annacurrey
as code owners
mjdenkowski
approved these changes
Sep 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context:
Pytorch has some bugs related to setting values for tensors to scalars when using torch.use_deterministic_algorithms(True) that were only addressed in most recent Pytorch versions. See:
The scalar attribution in beam_search.forward would throw:
when running with torch.use_deterministic_algorithms(True).
The change in the PR did not have any changes in performance, but works around the 2 bugs mentioned above.
Testing done:
torch.use_deterministic_algorithms(True)Pull Request Checklist
until you can check this box.
pytest)pytest test/system)./style-check.sh)sockeye/__init__.py. Major version bump if this is a backwards incompatible change.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.