Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

astrid: create integration commands to manually regenerate checkpoints, spans, events, etc. #12737

Open
taratorio opened this issue Nov 15, 2024 · 0 comments
Labels
astrid imp2 Medium importance polygon

Comments

@taratorio
Copy link
Member

relates to #12706 (comment)

We would like to be able to have a manual way to re-generate snapshot files for checkpoints, spans, state sync events using Astrid. Note to use Astrid standalone mode we use --polygon.sync=true (needs sync from scratch).

The flow would be something like:

  1. User will stop erigon
  2. To re-gen state sync events a user will
    • delete all snapshot files from datadir/snapshtos that relate to state sync events that we want to regenerate
    • rm -rf datadir/bridge to wipe off the Astrid Bridge DB
    • run ./build/bin/integration astrid_bridge_resync
  3. To re-gen spans, checkpoint, milestones a user will
    • delete all snapshot files from datadir/snapshtos that relate to checkpoints, spans, milestones that we want to regenerate
    • rm -rf datadir/heimdall to wipe off the Astrid Heimdall DB
    • run ./build/bin/integration astrid_heimdall_resync
  4. Start Erigon again which will cause it to go through the snapshots stage and that will retire the newly fetched data from the Bridge and Heidamll DBs

To create the astrid_bridge_resync command we need to:

  • initialise the polygon/bridge Bridge component
  • start it by calling bridge.Run in another goroutine (can be stopped by calling cancel on the run ctx)
  • call bridge.ProcessNewBlocks in batches of 5000 blocks to replay all block headers from genesis (block 0) to current latest canonical header (can use the block reader for that) - note can build empty block bodies (ie no need to fetch whole block from DB) as we just need the canonical header value (so can use types.NewBlockWithHeader)
  • ProcessNewBlocks has logic internally to wait for all events for each block to be fetched first so we do not have to worry about that at all
  • Once we've replayed all existing canonical blocks we're done - stop the bridge Run goroutine and exit

To create the astrid_heimdall_resync command we need to:

  • initialise the polygon/heimdall Service component
  • start it by calling heimdall.Run in another goroutine (can be stopped by calling cancel on the run ctx)
  • call heimdall.SynchronizeCheckpoints to wait for it to fetch all checkpoints
  • call heimdall.SynchronizeMilestones to wait for it to fetch all milestones
  • call heimdall.SynchronizeSpans to wait for it to fetch all spans
  • once the above 3 have finished we can stop the heimdall goroutine and exit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
astrid imp2 Medium importance polygon
Projects
None yet
Development

No branches or pull requests

1 participant