text-scale breaks scrolling

[TheRobotFox]

Topspace does not behave when changing the text scale with text-scale-set. When you scroll past the top the screen wraps around the point instead of moving it.
Also evil-scroll-* commands misbehave, sometimes the down scroll gets stuck at a certain point and stops moving further, while the up scroll will once again wrap the screen.
The problem occurs after changing the buffer's text-scale and persists even after reverting the change.

To Reproduce

1. Open Buffer
2. enable topspace
3. M-x eval-expression (text-scale-set 2)
4. scroll past the point
5. try evil-mode scroll movement

Emacs version

GNU Emacs 28.2 (build 2, x86_64-w64-mingw32) of 2022-09-13.