pageserver: refactor generation-aware loading code into generic

[jcsp]

Problem

Indices used to be the only kind of object where we had to search across generations to find the most recent one. As of #9543, manifests will need the same treatment.

Summary of changes

• Refactor download_index_part to a generic download_generation_object function, which will be usable for downloading manifest objects as well.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[arpad-m]

LGTM

[github-actions]

5274 tests run: 5052 passed, 0 failed, 222 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_isolation[None]: release-arm64

Postgres 15

• test_isolation[None]: release-arm64

Code coverage* (full report)

• functions: 31.3% (7690 of 24566 functions)
• lines: 48.7% (60429 of 124000 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
05de60b at 2024-10-29T10:14:37.010Z :recycle:}

[arpad-m]

Note that one can go further than this: the two manifests both implement deserialization using serde, and the download functions are identical except for the path generation. One could make the path generation function be the parameter, and have the same download code. This would counteract the refactor I did in #9444 to create a generic download function, but it can be undone as well.