super_deap OP t1_jckpoey wrote on March 17, 2023 at 3:17 PM

Reply to comment by harharveryfunny in [D] PyTorch 2.0 Native Flash Attention 32k Context Window by super_deap

I think one just needs to duplicate positional embeddings and we are good to go. Of course, there needs to be more comprehensive empirical analysis on this and I have not come across any of such attempts. I did a basic experiment and it seems to work but will have to wait and see.

Sad-Comedian-711 t1_jcqgv1x wrote on March 18, 2023 at 7:40 PM

This approach has been shown to work. Longformer even provided a script that did this for you: https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb

I think for flash attention you do not want to use Longformer's attention though, you want to use Big Bird's with specific block sizes or something like that.