Introducing RWKV - An RNN with the advantages of a transformer
What changed
The RWKV project was kicked off and is being led by Bo Peng, who is actively contributing and maintaining the project. Efforts have been made to address this limitation by introducing new architectures such as LSTMs or GRUs. Recently a new multilingual tokenizer has been released.
Why it matters
A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.
How it compares
Related prior coverage to compare against:
- Train your first Decision Transformer
- Differential Transformer V2
- Patch Time Series Transformer in Hugging Face
Sources
- Introducing RWKV - An RNN with the advantages of a transformer (huggingface-blog)primary