If your t/s is low enough, you can see speculative decoding with your own eyes
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
The other day I was trying out a distillation of DS4 Pro, and it came with MTP. It was slow as hell on my hardware, barely 2-3 t/s, BUT the speed got bumps every once in a while, and I noticed it was in moments like:
- United States of America
- First law of thermodynamics
- The enshittification of the internet
Basically, every time a very predictable phrase came up, it was instantly written. A fun thing to see.
But it also has me wondering - would MTP work together with n-grams? Since n-grams are Markov chains, the same engine behind autosuggest, how much sense would it make to combine them with speculative decoding?
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.