Which API for general real-time LLM agents [D]

TL;DR what is the right interface for writing custom real-time LLM agents?

I've noticed many projects (academic, startups, pet, etc) are trying to make LLMs work in real-time world where things happen while the model is thinking. And the agents need to be fast, but also have some way to new stimuli while they think, like when a user interrupts a voice agent, or browser pop-ups, etc. And these things already exist in principle, e.g. most robotics agents, real-time assistants [videoLLM] [gpt4o], video calls [wan-streamer], also [mid-turn steering] in Codex where you can add clarifications while the agent works, etc.

So it looks like LLMs can already do real-time interaction for specific scenarios --- I was wondering if it can be generalized so developers can write their own real-time agents as we write custom MCP tools now. So, we would need some sort of reusable building blocks that can be combined into, say, a coding agent that helps you debug in real time, or a "deep research" agent that talks to you and reacts to your feedback mid-search, or a voice-controlled calendar/spreadsheet helper, etc. A bit silly, my point is: there are applications not covered by current models.

I was wondering: if someone were to build this, what interface would you use for building async agents in general? Vendors technically have a "real-time APIs"[1,2]", but those are for coding. The closest thing I found about general async agents is [AsyncLLM preprint 2609.35427] (disclaimer: I know the authors), where the programmer uses asyncio to write coroutines with shared memory blocks, which is how agents communicate (image below from that preprint).

img from https://huggingface.co/papers/2609.35427

However, that still assumes the programmer writes a low-level inference pipeline, so perhaps there's a more natural way that would let openai/anthropic/etc expose some API that would let developers run general async agents for their use case and feed them with real-time streams of events.

[based on AsyncLLM, System 1 LLMs, interaction models, inference libs]