bo_peng OP t1_jb1po7i wrote
Reply to comment by _Arsenie_Boca_ in [R] RWKV (100% RNN) can genuinely model ctx4k+ documents in Pile, and RWKV model+inference+generation in 150 lines of Python by bo_peng
Will the 150 lines help? Please read the code first :)
https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_lines.py
This is ALL you need for RWKV inference.
And you can read https://arxiv.org/abs/2302.13939 (SpikeGPT) which is inspired by RWKV and has plenty of explanations :)
_Arsenie_Boca_ t1_jb1wjfi wrote
It does help but certainly doesnt make everything clear. I am confident I could run inference on it, but my interest is rather academic than practical.
What is the magic number 5 all about? It seems to appear all over the code without explanation.
Are the time mixing and channel mixing operations novel or were they introduced by a citable work?
How does the parallelization during training work?
bo_peng OP t1_jb1z3an wrote
5 is the number of hidden states per block (4 for ATT = xx aa bb pp, 1 for FFN = xx).
TimeMixing is RWKV.
ChannelMixing is your usual FFN (sqReLU as in Primer paper) with an extra R-gate (Novel. I find it helps).
Parallelization is due to https://github.com/BlinkDL/RWKV-LM/raw/main/RWKV-formula.png.
Viewing a single comment thread. View all comments