Are there any interesting architectural innovations that we seem to be on the verge of for LLM models or AI models that might be a big deal? (Excluding maybe N-gram, since everyone is already well aware of that one)

So, ideally for this thread we exclude the ones that everyone on here is already well aware of and discussing on here a lot, like N-gram, quantization improvements, MTP, D-flash, and D-spark, since those are improvement areas that most people on here are already pretty familiar with.

I'm more curious about any interesting fundamental architectural changes to either LLMs or other types of AI models, that you guys have been reading about or is starting to get any buzz that maybe most of us don't know about.

I know one person on here seemed pretty interested in the possibilities of more MAMBA-leaning architectures, although I don't know enough about AI to understand what makes it interesting compared to the more traditional LLM transformers and how they do attention. Like, what the high-end potential would be if people took it to greater extremes, let's say.

Anyway, I am curious if there are any other notable architectural things, maybe even more significantly different than just MAMBA or hybrid architecture changes, if there are some more radical ones you've seen people theorizing about, that maybe some of you have found interesting or think have a lot of potential.

And if possible, explain why you think it is interesting or might have a lot of potential.