Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mempool syncing optimizations & fixups #89

Merged
merged 7 commits into from
Oct 1, 2024

Commits on Sep 10, 2024

  1. Don't use RPC batching with bitcoind

    This actually hurts performance because the batched response has to be
    bueffered on the bitcoind side, as @TheBlueMatt explains at
    romanz#373 (comment)
    
    Instead, send multiple individual RPC requests in parallel using a
    thread pool, with a separate RPC TCP connection for each thread.
    
    Also see romanz#374
    shesek committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    5b07357 View commit details
    Browse the repository at this point in the history
  2. Don't store mempool txs before all prevouts are available

    The indexing process was adding transactions into the store so that
    prevouts funded & spent within the same batch could be looked up via
    Mempool::lookup_txos(). If the indexing process later failed for any
    reason, these transactions would remain in the store.
    
    With this change, we instead explicitly look for prevouts funded within
    the same batch, then look for the rest in the chain/mempool indexes and
    fail if any are missing, without keeping the transactions in the store.
    shesek committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    5c8c785 View commit details
    Browse the repository at this point in the history
  3. Continuously attempt to fetch mempool transactions

    Previously, if any of the mempool transactions were not available
    because they were evicted between getting the mempool txids and txs
    themselves, the mempool syncing operation would be aborted and tried
    again from scratch.
    
    With this change, we instead keep whatever transactions we were able to
    fetch, then get the updated list of mempool txids and re-try fetching
    missing ones continuously until we're able to get a full snapshot.
    shesek committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    4197301 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e3481c7 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Update logging verbosity

    - Reduce logging level for bitcoind's JSONRPC response errors
    
      These can happen pretty often for missing mempool txs and do not
      warrant warn-level logging. Unexpected RPC errors will bubble up and
      be handled appropriately.
    
    - Add more verbose logging for mempool syncing
    shesek committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ebcd737 View commit details
    Browse the repository at this point in the history
  2. Reuse RPC threads and TCP connections

    Keep RPC TCP connections open between sync runs and reuse them,
    to reduce TCP connection initialization overhead.
    shesek committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    aee32b6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7a068bf View commit details
    Browse the repository at this point in the history