Products & tools · first seen 1 Aug, updated 1 Aug
Speculative Decoding Collapses at Batch 224 on Llama-3-70B and Never on gpt-oss-120B
I built a 180-line roofline model of speculative decoding this week, and the number that fell out of it is batch 224. That is where… Continue reading on Towards AI »
Summary from Towards AI.