Inconsistent data between /block and /tx_search endpoints

[kinrokinro]

Bug Report

Setup

CometBFT version (use cometbft version or git rev-parse --verify HEAD if installed from source):

$ ~/.celestia-app/cosmovisor/current/bin/celestia-appd version --long 2>&1| grep core
- github.com/tendermint/[email protected] => github.com/celestiaorg/[email protected]

Have you tried the latest version: yes

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 20.04.3 LTS

What happened?

When querying /block and /tx_search, they return inconsistent data.
In this example /block returns 17 txs, but tx_search only 1:
https://celestia-testnet-rpc.itrocket.net/block?height=3115506
https://celestia-testnet-rpc.itrocket.net/tx_search?query="tx.height=3115506"

https://rpc.archive.celestia.testnet.dteam.tech/block?height=3115506
https://rpc.archive.celestia.testnet.dteam.tech/tx_search?query="tx.height=3115506"

What did you expect to happen?

All endpoints are expected to provide consistent data for this queries.

[mindstyle85]

hey @evan-forbes can you take a look at this please?

[kinrokinro]

Update: everything works fine in the previous mocha release v2.3.1-mocha, so I will revert our testnet RPC for now as a workaround.

[evan-forbes]

@kinrokinro @mindstyle85 we'll pick this up this week

[ninabarbakadze]

hey @kinrokinro

The tx search endpoint shows unexpected EOF in this file https://celestia-testnet-rpc.itrocket.net/tx_search?query="tx.height=3115506 was there a different log showing 1 transaction?

Did this issue occur on any other block height?

[kinrokinro]

hey @kinrokinro

The tx search endpoint shows unexpected EOF in this file https://celestia-testnet-rpc.itrocket.net/tx_search?query="tx.height=3115506 was there a different log showing 1 transaction?

Did this issue occur on any other block height?

You missed the quote, please try this:
https://celestia-testnet-rpc.itrocket.net/tx_search?query=%22tx.height=3115506%22

Yes, this issue occur on almost any block height (probably except blocks without txs)

[ninabarbakadze]

@kinrokinro @mindstyle85

TL;DR of the Issue:

In the kv.go file, the Search() function, used to retrieve transactions for a specific block height, had a bug where transaction hashes of type []byte were stored in a map without copying their data. Since Go handles []byte as a reference type, the map stored references to the same underlying memory for it.Value(), meaning when the iterator advanced and it.Value() updated, all map values were also updated, even though the keys remained unique. Additionally, Search() only allows unique hashes to be returned, so this caused the function to return just one transaction instead of all duplicate ones. The fix involved copying the []byte data into a new slice before storing it in the map. I tested the fix by running a node locally and tx_search endpoint returned all unique transaction from the block.

Next steps:

Could you please bump celestia-app to this commit and verify that the solution works - 4b0ba943a7369af45127a8e139cfc1fd72eb061f.

This code has been in the codebase for years and was not modified by us in v3. I’m still investigating why it broke in v3, but the upstream codebase contains the same bug. We will upstream this fix as well.

[kinrokinro]

@kinrokinro @mindstyle85

TL;DR of the Issue:

In the kv.go file, the Search() function, used to retrieve transactions for a specific block height, had a bug where transaction hashes of type []byte were stored in a map without copying their data. Since Go handles []byte as a reference type, the map stored references to the same underlying memory for it.Value(), meaning when the iterator advanced and it.Value() updated, all map values were also updated, even though the keys remained unique. Additionally, Search() only allows unique hashes to be returned, so this caused the function to return just one transaction instead of all duplicate ones. The fix involved copying the []byte data into a new slice before storing it in the map. I tested the fix by running a node locally and tx_search endpoint returned all unique transaction from the block.

Next steps:

Could you please bump celestia-app to this commit and verify that the solution works - 4b0ba943a7369af45127a8e139cfc1fd72eb061f.

This code has been in the codebase for years and was not modified by us in v3. I’m still investigating why it broke in v3, but the upstream codebase contains the same bug. We will upstream this fix as well.

It works, thanks!

$ curl -s localhost:$RPC_PORT/tx_search?query=\"tx.height=3139389\" | jq | grep total
    "total_count": "9"