[WIP]: dedup if-err-then-call-on_key_error-and-set-ignore_key_with_err-and-continue logic

[yliang412]

Problem

#8942 (comment)

Summary of changes

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[yliang412]

Dropping this.

[github-actions]

4977 tests run: 4811 passed, 2 failed, 164 skipped (full report)

---

Failures on Postgres 17

• test_pageserver_compaction_smoke: release-arm64

Failures on Postgres 14

• test_lfc_resize: release-arm64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_lfc_resize[release-pg14] or test_pageserver_compaction_smoke[release-pg17]"

Flaky tests (1)

Postgres 14

• test_location_conf_churn[1]: release-arm64

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
ee2c841 at 2024-09-24T15:04:11.415Z :recycle:}