Differential Transformer V2

What changed

We compare DIFF V2 with DIFF V1 below: (For simplicity, we omit the batch dimension and assume that both the input and output of the following flash_attn_func are three-dimensional tensors (tokens, heads, head dimension). DIFF V2 introduces additional query heads compared to the baseline Transformer, but does not increase the number of key-value (KV) heads. During pretraining, when using cutting-edge FlashAttention kernels on H-series and B-series GPUs, the throughput reduction introduced by DIFF V2 is negligible.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Train your first Decision Transformer
• Patch Time Series Transformer in Hugging Face
• Transformer-based Encoder-Decoder Models

• Train your first Decision Transformer similarity 0.22
• Patch Time Series Transformer in Hugging Face similarity 0.20
• Transformer-based Encoder-Decoder Models similarity 0.19