r/LocalLLaMA · · 1 min read

If your t/s is low enough, you can see speculative decoding with your own eyes

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

The other day I was trying out a distillation of DS4 Pro, and it came with MTP. It was slow as hell on my hardware, barely 2-3 t/s, BUT the speed got bumps every once in a while, and I noticed it was in moments like:

  • United States of America
  • First law of thermodynamics
  • The enshittification of the internet

Basically, every time a very predictable phrase came up, it was instantly written. A fun thing to see.

But it also has me wondering - would MTP work together with n-grams? Since n-grams are Markov chains, the same engine behind autosuggest, how much sense would it make to combine them with speculative decoding?

submitted by /u/zippydazoop
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA