A paper published on Hugging Face presents FlashRT, a system that uses a coding agent to optimize the development of multimodal real-time applications: voice agents, interactive video generation, pipelines where different models need to be chained across multiple GPUs.

The operational problem is well-known to anyone scaling these systems in production. Having good models isn’t enough: you need to decide where to run each component (placement), how to stream data between one model and the next (streaming), and how to parallelize within each individual model. Today these decisions are made by an expert by hand, for each application.

FlashRT delegates this work to a coding agent. It takes a reference implementation written by the developer, transforms it into an intermediate representation, validates it with an interpreter, and then iteratively tries different optimizations by measuring results on the hardware. According to the paper (published July 20, 2026), on NVIDIA B200 GPUs it achieves up to 70x latency reduction and 2.8x throughput improvement. On AMD MI355X, where expert optimization is less mature, it reaches 3.6x throughput and beats vLLM-Omni expert implementation by 65% on latency.

This continues a thread we’ve already discussed: in production agentic systems, orchestration matters more than the model. Here the idea goes further. The agent doesn’t just orchestrate runtime, it optimizes the deployment itself.

In detail

What came before.

Modern serving systems (vLLM, TensorRT-LLM and similar) optimize inference for a single model. When you combine heterogeneous models in a pipeline, like a voice agent does (ASR, LLM, TTS), optimization decisions become application-specific: where to place each model, how to stream data between them, how to parallelize each model internally. Existing auto-parallelism compilers rely on fixed assumptions about workload and limited transformations. Result: for each new application, an expert must rewrite the implementation by hand.

How FlashRT works.

FlashRT is a harness for coding agents. The process has four phases:

1. The agent transforms the reference code into an intermediate representation (IR) that captures data dependencies and persistent state scopes.
2. A sequential interpreter validates this IR: runs it to verify it does what it should.
3. The agent performs static analysis to identify candidate transformations (different ways to place, parallelize, stream data).
4. For each candidate, the agent implements, verifies, and measures in an iterative loop driven by concrete results (latency, throughput) on the target hardware.

The novelty lies in delegating to a generic coding agent work that previously required a systems expert for each application. The authors call this approach “chain-of-program”: the agent receives structured instructions on how to transform code, not free-form prompts.

The numbers in context.

Results reported by the paper:

• On NVIDIA B200: up to 70x latency reduction and 2.8x improved throughput, on applications including video world models and multimodal LLMs.
• On AMD MI355X: same peak latency reduction, 3.6x throughput. For Qwen3-Omni text-to-audio, 65% less latency than vLLM-Omni expert implementation.

The AMD figure is most interesting. On platforms where manual optimization is less mature, the agentic approach recovers more ground. It signals that agent-driven optimization scales better where years of cumulative expert work is lacking.

Limitations.

The paper came out days ago and has one upvote on Hugging Face Daily Papers: independent coverage is still minimal. Numbers are self-reported by authors, measured on specific hardware (B200, MI355X) not accessible to everyone. The repo is public on GitHub, but it’s unclear how straightforward reproducing results is outside the paper’s benchmarks. As always with systems papers, the real proof comes when others try it on their own pipelines.