Petals runs large language models locally by distributing weights across devices, BitTorrent-style. You load a chunk of the model onto your consumer GPU, connect to a network of people serving the other chunks, and use the full model.

The difference from a cloud API is control: you get access to the model in PyTorch, can fine-tune, follow custom paths, read hidden states. The difference from a solitary local installation is you don’t need a GPU cluster.

Open models keep growing in size, like the 2.8T parameter Kimi K3 released in July. Petals tackles the mirror problem: how to run them without a data center. The project website reports up to 6 tokens per second for Llama 2 70B and 4 for Falcon 180B, enough for an interactive chatbot.

The tradeoff is network dependency. If peers serving critical chunks go offline, the model slows down. And the reported speeds apply to single-batch inference, not heavy workloads.

If you want to try it

The Colab notebook linked on the Petals website is the fastest starting point, no local installation needed.

In detail

Local inference is taking different paths. Some projects quantize models to 1-bit to run them in a web page, shrinking the model until it fits on a single device. Petals goes the opposite direction: keeps the model at its original size and distributes the computational load.

Here’s how it works. A language model is a sequence of layers, each with its own weights. In a normal setup, all layers sit on the same machine. With Petals, each device on the network loads a group of layers. When you generate text, your GPU computes the first layers, passes the intermediate result to the next device, which computes its layers, and so on to the end. The final result comes back to you.

It’s the same principle as BitTorrent, but applied to computation instead of file transfer: no one has the whole model, but the network together serves it all.

The practical benefits go beyond text generation. Because you get access to the model at the PyTorch and Hugging Face Transformers level, you can apply custom sampling methods, run fine-tuning, or inspect hidden states. On a cloud API these things aren’t available: you just get the generated text back. For researchers or anyone experimenting with post-training techniques, having a model with hundreds of billions of parameters accessible at this level opens up different kinds of experiments.

The limits are real. Latency depends on your peer network: if whoever’s serving critical layers disconnects, the model slows or freezes. The speeds reported on Petals’ website (6 tokens/sec for Llama 2 70B, 4 for Falcon 180B) apply to single-batch inference, one request at a time. It’s not designed to serve concurrent traffic, and network dependency makes it unsuitable for stable production loads.

Security is an open question. You trust that network peers compute the weights correctly: a malicious peer could alter intermediate results without you knowing. It’s a different tradeoff than a cloud API, where the provider guarantees computation integrity.

Petals is part of the BigScience research workshop, the project that produced BLOOM (176B), one of the first large open models. The distributed approach makes sense at a moment when open-weight models are growing rapidly: when the largest open model goes from 176B to 2.8T parameters in a few years, distributing the load becomes a real necessity for anyone wanting to run them privately.

For those ready to accept network fragility in exchange for data control, Petals is an option that didn’t exist before. For those needing stability and throughput, cloud APIs remain the right choice.