Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block builder scheduler startup: learn state from worker updates #9897

Open
wants to merge 28 commits into
base: main
Choose a base branch
from

Conversation

seizethedave
Copy link
Contributor

@seizethedave seizethedave commented Nov 13, 2024

What this PR does

More progress on the block-builder-scheduler:

  • Add a startup period where we're learning the state of the world. This includes:
    • committed offsets learned from Kafka
    • any job progress/completion updates we've learned from workers given block-builder-scheduler.startup-observe-time
  • When the startup period if over, we compute the starting state from the observations and begin normal operation.
  • Add a job assignment epoch mechanism to break ties when multiple workers report updates about the same job. Clients include the epoch in any job-related communications.

What's next?

  • Flush updated offsets periodically back to Kafka.
  • RPC + client layers.

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@seizethedave seizethedave marked this pull request as ready for review November 13, 2024 23:26
@seizethedave seizethedave requested a review from a team as a code owner November 13, 2024 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant