Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update from Master #1197

Merged
merged 519 commits into from
Nov 8, 2024
Merged

Update from Master #1197

merged 519 commits into from
Nov 8, 2024

Conversation

maxpiva
Copy link
Member

@maxpiva maxpiva commented Nov 8, 2024

No description provided.

revam and others added 30 commits August 27, 2024 07:31
Because I got complaints about my old school nullable value checks being unreadable to others. 😔
…requency (#1172)

* Add setting for UDP ping frequency

* Reduce UDP logout period to 5 mins

* Fix ping timer calculation

* Retry UDP commands which failed with "UNKNOWN COMMAND"

* Change warning text for UNKNOWN COMMAND
- Reworked the internals to use the AniDB episode on the AniDB side when adding/removing AniDB ↔ TMDB movie links.

- Added new endpoints to link and unlink using the AniDB episode ID.

- Made is possible to also unlink a single episode when unlinking from the series link endpoint.

- Updated casing in log statements in the linking service.

- Fixed the purge logic for movies to work the same as the (previously fixed) show purge logic.
in export if comments are enabled.
…adata

This will prevent the same metadata entity from being updated concurrently, and will also ensure that a purge and update of the same metadata entity are not ran concurrently, which would be catastrophical.
revam and others added 28 commits October 27, 2024 00:53
Added better support for multi-episode files in playlists by changing the structure from 1-N to N-N, but we restrict playlist items to a 1-N or N-1 pattern even if the playlist theoretically allows N-N items.

Tweaked the playlist hydration to be more flexible, and moved the hydration logic from the API layer to the service layer.

Series and episodes now use AniDB IDs, while files use either local IDs or lookup by ED2K hash, optionally with a file size provided.

The query options have all been moved into the playlist near their respective item, instead of being applied globally, so we can apply the series options on a per series basis and preferred release group ID per series and/or episode.

Because of the new flexibility then we can now provide episodes and files together to to watch one or more files for one or more episodes, instead of leaving the file/episode choice up to the auto-detect logic. See the example below for a made up example of how this would work

**Example**

Given a dehydrated playlist as follows;

```txt
api/v3/Playlist/Generate?playlist=e2345+f234234,e2346,e953423+20A3246CC1BD7C513598FD49F250348B,3456,s69+onlyUnwatched+includeSpecials,e23423+e953423+20A3246CC1BD7C513598FD49F250348B-135345345345,s12+r123+includeRewatching
```

It would hydrate to the following:

- `e2345+f234234`: Add episode `2345` with local file `234234`.

- `e2346`: Add episode `2346` without a file specified, letting the system auto-detect the file to use.

- `e953423+20A3246CC1BD7C513598FD49F250348B`: Add episode `953423` with the local file specified by the ED2K hash `20A3246CC1BD7C513598FD49F250348B`.

- `3456`: Add local file `3456` without an episode specified, letting the system auto-detect the episode to use.

- `s69+onlyUnwatched+includeSpecials`: Add unwatched episodes and specials from series `69`.

- `e23423+e953423+20A3246CC1BD7C513598FD49F250348B-135345345345`: Add the local file specified by the ED2K hash `20A3246CC1BD7C513598FD49F250348B` and file size `135345345345` for episode `23423` and `953423`.

- `s12+r123+includeRewatching`: Prefer release group `123` for series `12` and start the playlist from the first episode that's being re-watched if any, or otherwise fallback to the normal behavior.
Since the uri will only set the port if it's needed while the builder will always include the port no matter what.
Remove the stall entry for a file that does not exist anymore if we're attempting to move to a new location, but we have a file location entry (VideoLocal_Place entry) in the database but there is no file at the location in the file system, then proceed with the relocation after the previous entry has been removed.
because it doesn't require a body and I need it to be get for the vlc scrobbler.
after the initial burst of commands to hopefully prevent API bans.

The long delay for UDP has been changed from two times the short delay to three times (4s → 6s), and HTTP has been changed from two times to fifteen times (4s → 30s). To compensate, then the reset period is down from 3m to 2m.

If these numbers are too preservative then we can tweak them again.

Related to #1182
Switch from using a monitor and thread sleep (both sync) while rate limiting to using a semaphore slim and thread delay (both async) to not block the thread while we wait for the delay to elapse.
we still might want to keep them in the db in case the file is ever added back, but we don't want it to show up in the normal APIs.
Attempt to match by title plus year first then only title afterwards, instead of alternating the two 3 times in the TMDB auto match series search logic.
Make the AniDB rate limiting configurable and tweak the defaults (again) to be less preservative for HTTP.

Also cached the values per rate limiter (UDP and HTTP) and added cache invalidation when the settings has been saved to re-apply the (potentially) updated settings onto the locally cached values.
…ms and file systems

It's better, but still not perfect.

We now check the source/destination if they're case sensitive by probing the file system, and the API endpoints will use a less accurate check by default since the more accurate check requires probing, which we don't want to do in parallel on thousands of files unless told to do it.

Also consolidated the `IsLinux` and `IsRunningOnLinuxOrMac()` helpers into a single `IsLinuxOrMac` helper.
and instead allow the missing id to be later retrieved.
…s across platforms and file systems"

This reverts commit ef1a16c.

It was too early and causing other applications strain. Might re-implement it a different way in the future. Maybe.
Added two new match rating for when the title kinda matches but not exactly, one for only title and one for when the data also matches. Then added a new second pass to check for exact title matches after the first pass that checks for date and exact title matches, and moved the existing second pass to a third pass instead. This should make the matching _even more_ accurate, but it's still not perfect, and also getting more compute intense for every new tweak we're doing. In the worst case scenario then we just went from O(n²) to O(n³), but the best case scenario still remains O(n¹).
@maxpiva maxpiva merged commit 409120f into PostgreSQL Nov 8, 2024
15 checks passed
Copy link

sentry-io bot commented Nov 12, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ NHibernate.Exceptions.GenericADOException: could not execute query GET Group.GetGroup View Issue
  • ‼️ NHibernate.Exceptions.GenericADOException: could not execute query GET Series.GetSeriesImages View Issue
  • ‼️ NHibernate.Exceptions.GenericADOException: could not delete: [Shoko.Server.Models.SVR_AnimeEpisode#22551][SQL: DELETE FROM AnimeEpisode WHER... DELETE Series.DeleteSeries View Issue
  • ‼️ NHibernate.Exceptions.GenericADOException: could not execute query GET Tree.GetSeriesInFilteredGroup View Issue
  • ‼️ NHibernate.Exceptions.GenericADOException: could not execute query GET Tree.GetSeriesInFilteredGroup View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants