Usage of archival RPC for fetching versioned components

[mpeterdev]

We have work in progress to allow specifying the block height version of a component to load. If the version is not highly recent, then the fetch cannot be resolved by a standard RPC node.

The previous BOS VM handles this by sending all component fetches which specify a blockheight to the near.org archival RPC node.

We will be making a functional change to BWE where a significantly higher portion of component fetches will have a block height specified since the default behavior will be to lock child component versions based on their parent's publish block height.

Two main questions:

• Are the archival nodes equipped to handle this additional traffic?
• How much does this impact decentralization? We have been operating under the assumption that users would be able to specify their RPC provider but I am not information on any other providers which offer archival nodes

[ecp88]

We can accomodate the load. How much load are we talking about here?
You need to set a referer header to something we can whitelist you in the load balancer, if not, you will be throttled.
have you tested this with some test requests to archival? do you get the response you expect?

[mpeterdev]

We can accomodate the load. How much load are we talking about here?

Unfortunately I don't have info on what the expected usage of decentralized frontends has been estimated at in the past. @charleslavon do you know if we have data around this? Maybe we can use our existing analytics for near.org

You need to set a referer header to something we can whitelist you in the load balancer, if not, you will be throttled.

Sounds good, I will start looking into how to specify that with the nearjs lib we use to connect to the RPC

have you tested this with some test requests to archival? do you get the response you expect?

The previous BOS VM has been functioning this way for a while so I am not expecting any surprises around getting the data we need, but we haven't tested this new implementation with it yet

[ecp88]

We can accomodate the load. How much load are we talking about here?

Unfortunately I don't have info on what the expected usage of decentralized frontends has been estimated at in the past. @charleslavon do you know if we have data around this? Maybe we can use our existing analytics for near.org

We have capacity for a ~2000 req/s increase. If we are talking less than this for the first month then we are good.

[mpeterdev]

@ecp88 is it correct that we are the only archival node provider?

[ecp88]

@ecp88 is it correct that we are the only archival node provider?

I dont know if lava is archival https://www.lavanet.xyz/get-started/near
But assume we are the only ones.

[charleslavon]

@mpeterdev should BWE use RPC by default, or our QueryAPI components indexer? Your question about decentralization is a good one. Perhaps we add RPC/QueryAPI configs into the BWE initialization method to allow devs to choose their preferred provider.

hey @s-n-park we could use your help to get an idea of the frequency of requests made to URLs of the form (near.social/near.org)//widget/

[mpeterdev]

We've considered moving to QueryAPI in the long run but haven't pulled the trigger because that merits deeper discussion around both reliability and decentralization. It sounds like heavy reliance on an archival node may put this solution closer to equal footing with QueryAPI though

[mpeterdev]

hey @s-n-park we could use your help to get an idea of the frequency of requests made to URLs of the form (near.social/near.org)//widget/

This would be helpful but only a partial picture since a page load consists of potentially numerous component fetches. Do we have analytics on our RPC nodes that we could look at SocialDB get calls?

[s-n-park]

@charleslavon , @mpeterdev - looks like the rates are for get calls for social.near since 3/1/24 are:

1. Daily: ~1M
2. Hourly: roughly between 40k-60k
3. Minute: roughly between 500-2k

Hourly chart

Query

Per Minute chart:

I also checked to see if req/s spiked >2k and that only seemed to happen 1X since 3/1/24.

[charleslavon]

thank you @s-n-park

[charleslavon]

@mpeterdev Pavel's Q2 objective for QueryAPI is to be the preferred solution for building custom indexers for Near. We should work with them to build QueryAPI into BWE to minimize the custom code necessary to fetch data from one of our core indexers. It's also worth considering the impact on developer UX & UI performance of using RPC or QueryAPI as default for archival data needs.