Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pageserver: add supplementary branch usage stats #8131

Merged
merged 1 commit into from
Jul 3, 2024
Merged

pageserver: add supplementary branch usage stats #8131

merged 1 commit into from
Jul 3, 2024

Conversation

jcsp
Copy link
Collaborator

@jcsp jcsp commented Jun 21, 2024 •
edited
Loading

Problem

The metrics we have today aren't convenient for planning around the impact of timeline archival on costs.

Closes: #8108

Summary of changes

  • Add metric pageserver_archive_size, which indicates the logical bytes of data which we would expect to write into an archived branch.
  • Add metric pageserver_pitr_history_size, which indicates the distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and associated consumption metric changes, these will reach a final form. For now, an "archived" branch is just any branch outside of its parent's PITR window: later, archival will become an explicit state (which will usually correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are removing many more in #8245 than this PR is adding.

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@jcsp jcsp added c/storage/pageserver Component: storage: pageserver a/tech_debt Area: related to tech debt labels Jun 21, 2024
@jcsp jcsp changed the title pageserver: add supplementary branch usag stats pageserver: add supplementary branch usage stats Jun 21, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 21, 2024 •
edited
Loading

3006 tests run: 2891 passed, 0 failed, 115 skipped (full report)

Flaky tests (1)

Postgres 15

  • test_scrubber_tenant_snapshot[4]: release

Code coverage* (full report)

  • functions: 32.7% (6940 of 21218 functions)
  • lines: 50.0% (54369 of 108634 lines)

* collected from Rust tests only

The comment gets automatically updated with the latest test results
5fdfb60 at 2024-07-03T15:41:29.146Z :recycle:

@jcsp jcsp force-pushed the jcsp/branch-usage-metrics branch from 4187ad8 to 3eb5155 Compare July 3, 2024 12:42
@jcsp jcsp force-pushed the jcsp/branch-usage-metrics branch from 3eb5155 to 5fdfb60 Compare July 3, 2024 14:37
@jcsp jcsp marked this pull request as ready for review July 3, 2024 14:38
@jcsp jcsp requested a review from a team as a code owner July 3, 2024 14:38
@jcsp jcsp requested a review from skyzh July 3, 2024 14:38
skyzh
skyzh approved these changes Jul 3, 2024
View reviewed changes
Copy link
Member

@skyzh skyzh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

pageserver/src/tenant.rs Show resolved Hide resolved
@jcsp jcsp merged commit 778787d into main Jul 3, 2024
63 of 64 checks passed
@jcsp jcsp deleted the jcsp/branch-usage-metrics branch July 3, 2024 21:29
@jcsp jcsp mentioned this pull request Jul 3, 2024
Open
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
754e5d3 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
841ef39 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
f3c21f4 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
cb1b35b 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
dad88b0 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
@jcsp @VladLazar
pageserver: add supplementary branch usage stats (#8131)
7b7d16f 
## Problem

The metrics we have today aren't convenient for planning around the
impact of timeline archival on costs.

Closes: #8108

## Summary of changes

- Add metric `pageserver_archive_size`, which indicates the logical
bytes of data which we would expect to write into an archived branch.
- Add metric `pageserver_pitr_history_size`, which indicates the
distance between last_record_lsn and the PITR cutoff.

These metrics are somewhat temporary: when we implement #8088 and
associated consumption metric changes, these will reach a final form.
For now, an "archived" branch is just any branch outside of its parent's
PITR window: later, archival will become an explicit state (which will
_usually_ correspond to falling outside the parent's PITR window).

The overall volume of timeline metrics is something to watch, but we are
removing many more in #8245
than this PR is adding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skyzh skyzh skyzh approved these changes

Assignees
No one assigned
Labels
a/tech_debt Area: related to tech debt c/storage/pageserver Component: storage: pageserver
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Per-timeline metrics for whether branch is from within PITR interval, and its history size
2 participants
@jcsp @skyzh