Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize txo lookup #76

Merged
merged 3 commits into from
Aug 13, 2024
Merged

Optimize txo lookup #76

merged 3 commits into from
Aug 13, 2024

Conversation

shesek
Copy link
Collaborator

@shesek shesek commented Mar 25, 2024 •
edited
Loading

From https://github.com/facebook/rocksdb/wiki/MultiGet-Performance:

There is a lot of complexity in the underlying RocksDB implementation to lookup a key. The complexity results in a lot of computational overhead, mainly due to cache misses when probing bloom filters, virtual function call dispatches, key comparisons and IO. Users that need to lookup many keys in order to process an application level request end up calling Get() in a loop to read the required KVs.

By providing a MultiGet() API that accepts a batch of keys, it is possible for RocksDB to make the lookup more CPU efficient by reducing the number of virtual function calls and pipelining cache misses. Furthermore, latency can be reduced by doing IO in parallel.

@shesek shesek force-pushed the 202403-txo-lookup-optimizations branch 2 times, most recently from 0bbc5f5 to 5913e22 Compare March 25, 2024 19:15
@jamesdorfman
Copy link

These optimizations make sense to me.
utACK 5913e22.

@shesek shesek force-pushed the 202403-txo-lookup-optimizations branch 2 times, most recently from 61823dd to 5e27752 Compare March 26, 2024 18:21
@shesek shesek force-pushed the 202403-txo-lookup-optimizations branch from 5e27752 to a0d034d Compare May 22, 2024 23:31
@shesek
Optimize lookup_txos using rocksdb's MultiGet
168e241 
See https://github.com/facebook/rocksdb/wiki/MultiGet-Performance
@shesek
Optimize and refactor lookup_txos()
e9d68b2 
- Avoid unnecessary copying of prev outpoints

- When looking for both mempool and on-chain txos, accumulate the set of
  outpoints that remain to be looked up to avoid re-checking for them
  later again in the found set

- Refactored lookup_txos() to use lookup_txo() internally rather than
  the other way around, which was less efficient

- Lookup txos in mempool first, then on-chain

- ChainQuery::lookup_txos() now returns a Result instead of panicking
  when outpoints are missing

- Removed ChainQuery::lookup_avail_txos() and allow_missing, which are
  no longer neceesary
@shesek shesek force-pushed the 202403-txo-lookup-optimizations branch from a0d034d to e9d68b2 Compare May 30, 2024 04:03
@shesek
Copy link
Collaborator Author

shesek commented May 30, 2024

Rebased.

@shesek shesek mentioned this pull request Jun 1, 2024
Merged
@RCasatta
Copy link
Collaborator

RCasatta commented Aug 9, 2024

utACK 4f037c4

@RCasatta
Copy link
Collaborator

Needs rebase

@philippem philippem merged commit fca4867 into Blockstream:new-index Aug 13, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RCasatta RCasatta RCasatta approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shesek @jamesdorfman @RCasatta @philippem