Mesh LLM is a system that runs distributed language models on a peer-to-peer network built on iroh. Instead of buying larger GPUs or paying for a centralized API, it pools the machines you have — laptops, office servers, boxes under your desk — and makes them work as a single resource. Each node exposes an OpenAI-compatible endpoint on localhost:9337/v1, so existing clients work without changes.

When a request arrives, the system decides autonomously: run it locally if the model is already loaded, forward it to a peer that has it ready, or split the model across layers on multiple machines if it’s too big for one. This third scenario — called Skippy internally — lets you run models with 235 billion parameters on modest hardware, passing activations from one node to the next like a pipeline.

Why it matters to you. If you’re evaluating self-hosting for cost control, privacy, or model stability, this is a concrete approach: no lock-in, no vendor changing the model or prices on you. The catalog starts with over 40 ready-to-use models, from half a billion parameters up. The operational side — NAT traversal, public-key authentication, relays — is handled by iroh, which already runs in production on hundreds of thousands of devices.

If you want to try it. The software weighs 18 MB. Install, launch a node, and point any OpenAI client to http://localhost:9337/v1. To set up a private network instead of joining the public mesh, documentation is on the Mesh LLM site. A mobile app is on the way, built on iroh’s Swift SDK.

In depth

What came before

Large models run on large GPUs, and large GPUs live in someone else’s datacenter. You can use them via API, but you lose control: you don’t know when the model changes, you don’t decide where your data goes, and the bill grows with usage with no way to bring it down. Self-hosting means buying hardware that often exceeds the single use case, or accepting smaller models.

Distributed approaches already existed (FlexGen, Petals), but they required complex configurations or only worked in academic settings. Mesh LLM aims to make it operational: true peer-to-peer, no central coordinator, with authentication and NAT traversal solved underneath.

How it actually works

Each node starts an iroh endpoint, identified by a public key. Iroh is a networking library that opens direct, authenticated QUIC connections between devices anywhere they are, even behind NAT. If the direct connection fails, two public relays in different regions provide a path.

On top of QUIC, Mesh LLM builds three ALPN channels:

• mesh-llm/1: the main channel, carrying gossip (which models each node has, available GPU, latency), tunneled HTTP requests to peers, and control flows.
• mesh-llm-control/1: the control plane for private mesh owners (config sync, ownership attestations).
• skippy-stage/2: the low-latency channel that carries activations between stages of a split model.

Within the main channel, each QUIC stream has an initial byte that says what message type it carries: gossip (0x01), HTTP tunnel for inference (0x04), route request (0x05), dead peer notification (0x06), and so on. A single connection demultiplexes everything.

When a model is too large for one machine, Mesh LLM partitions it by layer range: layers 0–15 on one node, 16–31 on another, to the end. Activations flow from one stage to the next. The client sees no complexity: it always calls localhost:9337/v1/chat/completions.

How mature is it

The project scored 198 points on Hacker News and 44 comments on launch day, a signal of traction in the technical community. Iroh, the underlying library, already runs in production on hundreds of thousands of devices, so the networking part isn’t an experiment. Mesh LLM itself is new: the mobile app is still missing, and ACP support (the emerging agent protocol) is announced but not yet available.

Practical limits depend on latency between nodes: splitting a model across distant machines slows inference. The public mesh introduces the risk of untrusted peers, though each node authenticates the others by key. Private meshes solve this but require setup.

Where it fits

Mesh LLM answers a question that keeps coming back: how do you do self-hosting without buying entire racks? The answer is to pool what you have. It’s not the first peer-to-peer implementation for LLMs, but it’s the first built on top of an already battle-tested networking library and offering immediate compatibility with OpenAI clients. It’s a concrete alternative if control over your models matters more than the convenience of a managed API.

For teams building agents or automations that need to run locally, or for groups wanting to share internal GPUs without exposing data to a third party, this is a path worth exploring. For those optimizing for minimum latency or wanting uptime guarantees, a centralized API remains more suitable.