pytorch model & ring attention

[LzhinFdu]

Thanks for sharing this excellent great work. We want to use pytorch models to try the effect of ring attention. Are there any plans to develop ring attention implementation under pytorch?

[apexspyche]

What do you have in mind? Is this model suitable for tokenized ecosystem and bridging liquidity and creating a smart algorithm for bridging / blending / mending and growth hacking liquidity across and between multiple TOKENS

[kabachuha]

Lucidrains has a pytorch implementation of RingAttention https://github.com/lucidrains/ring-attention-pytorch

[LzhinFdu]

Lucidrains has a pytorch implementation of RingAttention https://github.com/lucidrains/ring-attention-pytorch

Have you tried this repo? I don’t know whether the experimental results are as expected.
Seems that the model posted on huggingface cannot use it directly to call ring attention.

[LzhinFdu]

What do you have in mind? Is this model suitable for tokenized ecosystem and bridging liquidity and creating a smart algorithm for bridging / blending / mending and growth hacking liquidity across and between multiple TOKENS

I just want to call ring attention when using the trained pytorch LLM for inference.