A spec sheet tells you what a box is supposed to do. This tells you what yours does, on your desk, tonight. Prefill against prompt length, throughput over a long generation, what happens when two people use it at once, and what a reasoning model charges for the tokens nobody reads.
Every number below came off a Tiiny Pocket running TiinyOS 0.1.29, out of the run's own saved JSON. Nothing here is illustrative.
Fire one request, then two, then four, then eight. Aggregate throughput does not move: 23.5, 24.4, 24.2, 24.2 tokens a second. Wall time doubles at every step, 6.8s to 52.9s.
Eight callers do not each get a slower stream. They get the same speed, in turn. We measured this on three different models and got the same flat line every time.
Plan for one inference at a time. It is the single most useful thing this tool has told us.
A 72-token prompt ingests at 135 tokens a second. A 6,260-token prompt ingests at 756. The per-token cost of context falls by more than five times as the prompt grows.
That is the opposite of the instinct most people bring from cloud APIs, where a longer prompt is simply a bigger bill. Here, feeding it a whole document is close to free compared with feeding it a sentence.
The first version read NPU utilisation once the request came back, which only ever caught the box going idle. It faithfully reported 0% under load while a 35B model was flat out.
Sampled on a thread during the generation it reads a median of 86.7% and a peak of 95.3% over 48 samples. Same box, same model, same minute.
We mention it because a benchmark that cannot catch its own errors has no business reporting yours.
It reads your box and lists everything on it. Tick what to measure, and it tells you what that will cost you before you press anything: how long it will take, and whether it is going to load and unload models or leave your box exactly as it found it.
A cloud benchmark ranks models by what they cost to call. You own this box, so the currency is the hundred NPU units you have and the fact that it runs one inference at a time. Every ranking here carries a tokens per second per unit column, and there is a page that answers the question you actually ask at three in the afternoon: can I run these two things at the same time?
Each class is scored on the figure its own users care about. Ranking a text model against a voice on one axis would be measuring nothing.
Four prompt lengths from 72 to 6,260 tokens, each asking for a four-token answer so decode barely registers. What a long prompt costs to read.
One unbroken 1,500-token generation with utilisation sampled while it runs. Short bursts flatter a box; this is the number that holds.
One, two, four and eight identical requests at once from separate threads. Aggregate against per-stream.
The same question asked with thinking off and then on. The ratio is wall time, because that is what a person waits.
Three prompts at 512×512, eight steps. Our box does a plate in 7.93 seconds, median.
Seconds of audio produced per second of clock, read out of the returned WAV header rather than guessed from a character count. Above 1.0 it talks faster than you listen.
One, eight and thirty-two at a time, with the vector width it hands back.
Tick what you want resident together and see whether it fits in the hundred before you find out the hard way.
No quality evaluation of any kind. Nothing here says a model is good, only how fast it is. A fast wrong answer is still wrong.
And by default it loads nothing, unloads nothing, and benchmarks whatever is already running. That is the property that lets you run it on a box doing real work. The two flags that do move models say so in plain words before they start, and put the box back the way they found it.
TiinyBench lives on tiinyapp.farm, the community app farm for the Pocket Lab. The farm handles the install, checks the release against its published checksum, and shows you what every app needs and asks for before you plant it.
farm install tiiny-bench
export TIINY_HOST=192.168.1.50export TIINY_KEY=…
Host only, no scheme and no port.
tiiny-bench --selfcheck
Five checks in the order they actually fail.
Exit code 0 or 1, so you can gate on it.
tiiny-bench --serve
Then localhost:8425. Or skip the app
and drive it from the command line.