TiinyBench

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 11.3x the wait, 0.94s at 68 tokens against 10.59s at 6,256. Reading got cheaper per token as the prompt grew, from 74 to 447 tok/s. Decode stayed near 37 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.02x the aggregate of one, and wall time grows in step with the callers, 8s to 65s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.1s and 0 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

50x the prompt cost 8.2x the wait, 1.05s at 127 tokens against 8.57s at 6,315. Reading got cheaper per token as the prompt grew, from 127 to 581 tok/s. Decode stayed near 16 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.07x the aggregate of one, and wall time grows in step with the callers, 9s to 68s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 10.4s sooner and used 197 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 19 against 19 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 11.7x the wait, 0.57s at 72 tokens against 6.70s at 6,260. Reading got cheaper per token as the prompt grew, from 131 to 740 tok/s. Decode stayed near 46 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.06x the aggregate of one, and wall time grows in step with the callers, 8s to 35s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time. 4 request(s) failed and are not in these figures.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 10.4x the wait, 0.67s at 72 tokens against 6.92s at 6,260. Reading got cheaper per token as the prompt grew, from 115 to 718 tok/s. Decode stayed near 23 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.17x the aggregate of one, and wall time grows in step with the callers, 7s to 49s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 3.0s and 93 tokens to the same question. The decode rate barely moved, 50 against 48 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 12.4x the wait, 0.56s at 72 tokens against 6.89s at 6,260. Reading got cheaper per token as the prompt grew, from 134 to 719 tok/s. Decode stayed near 55 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.07x the aggregate of one, and wall time grows in step with the callers, 7s to 55s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 5.7s and 158 tokens to the same question. The decode rate barely moved, 25 against 25 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 11.7x the wait, 0.58s at 72 tokens against 6.81s at 6,260. Reading got cheaper per token as the prompt grew, from 128 to 695 tok/s. Decode stayed near 55 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.03x the aggregate of one, and wall time grows in step with the callers, 6s to 46s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 4.2s and 127 tokens to the same question. The decode rate barely moved, 29 against 29 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

96x the prompt cost 20.2x the wait, 0.81s at 65 tokens against 16.46s at 6,253. Reading got cheaper per token as the prompt grew, from 83 to 286 tok/s. Decode stayed near 22 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.11x the aggregate of one, and wall time grows in step with the callers, 12s to 86s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 30.4s and 494 tokens to the same question. The decode rate barely moved, 16 against 16 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 8.6x the wait, 1.27s at 72 tokens against 11.00s at 6,260. Reading got cheaper per token as the prompt grew, from 61 to 453 tok/s. Decode stayed near 10 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.18x the aggregate of one, and wall time grows in step with the callers, 19s to 128s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 16.7s sooner and used 359 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 20 against 18 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 10.4x the wait, 1.27s at 72 tokens against 13.13s at 6,260. Reading got cheaper per token as the prompt grew, from 62 to 480 tok/s. Decode stayed near 10 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.11x the aggregate of one, and wall time grows in step with the callers, 16s to 121s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 0.8s and 48 tokens to the same question. The decode rate barely moved, 22 against 22 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 10.5x the wait, 1.24s at 72 tokens against 12.95s at 6,260. Reading got cheaper per token as the prompt grew, from 62 to 486 tok/s. Decode stayed near 13 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.16x the aggregate of one, and wall time grows in step with the callers, 30s to 209s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 5.6s sooner and used 30 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 6 against 6 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 18.7x the wait, 0.27s at 68 tokens against 4.96s at 6,256. Reading got cheaper per token as the prompt grew, from 305 to 960 tok/s. Decode stayed near 21 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.00x the aggregate of one, and wall time grows in step with the callers, 6s to 50s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 0.8s and 48 tokens to the same question. The decode rate barely moved, 51 against 52 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 17.2x the wait, 0.34s at 68 tokens against 5.93s at 6,256. Reading got cheaper per token as the prompt grew, from 209 to 799 tok/s. Decode stayed near 40 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.05x the aggregate of one, and wall time grows in step with the callers, 7s to 52s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 1.6s sooner and used 33 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 26 against 26 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 20.5x the wait, 0.23s at 68 tokens against 4.82s at 6,256. Reading got cheaper per token as the prompt grew, from 311 to 978 tok/s. Decode stayed near 48 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.03x the aggregate of one, and wall time grows in step with the callers, 5s to 43s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.5s and 9 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 22.4x the wait, 0.22s at 68 tokens against 5.01s at 6,256. Reading got cheaper per token as the prompt grew, from 327 to 941 tok/s. Decode stayed near 48 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.01x the aggregate of one, and wall time grows in step with the callers, 5s to 43s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.4s and 8 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

89x the prompt cost 20.3x the wait, 0.24s at 70 tokens against 4.88s at 6,258. Reading got cheaper per token as the prompt grew, from 324 to 968 tok/s. Decode stayed near 31 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.05x the aggregate of one, and wall time grows in step with the callers, 6s to 43s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.3s and 0 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

85x the prompt cost 15.1x the wait, 0.25s at 74 tokens against 3.77s at 6,262. Reading got cheaper per token as the prompt grew, from 326 to 1,254 tok/s. Decode stayed near 42 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.01x the aggregate of one, and wall time grows in step with the callers, 7s to 58s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 16.4s and 357 tokens to the same question. The decode rate barely moved, 22 against 22 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

95x the prompt cost 16.5x the wait, 0.27s at 66 tokens against 4.42s at 6,254. Reading got cheaper per token as the prompt grew, from 272 to 1,068 tok/s. Decode stayed near 35 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.07x the aggregate of one, and wall time grows in step with the callers, 6s to 42s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 2.1s sooner and used 79 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 32 against 32 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 8.3x the wait, 0.44s at 72 tokens against 3.67s at 6,260. Reading got cheaper per token as the prompt grew, from 172 to 1,356 tok/s. Decode stayed near 41 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.06x the aggregate of one, and wall time grows in step with the callers, 8s to 64s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 11.7s and 247 tokens to the same question. The decode rate barely moved, 21 against 21 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 21.0x the wait, 0.16s at 72 tokens against 3.34s at 6,260. Reading got cheaper per token as the prompt grew, from 533 to 1,419 tok/s. Decode stayed near 35 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.03x the aggregate of one, and wall time grows in step with the callers, 8s to 61s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 14.3s and 302 tokens to the same question. The decode rate barely moved, 21 against 21 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 21.9x the wait, 0.14s at 68 tokens against 2.98s at 6,256. Reading got cheaper per token as the prompt grew, from 584 to 1,590 tok/s. Decode stayed near 34 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.04x the aggregate of one, and wall time grows in step with the callers, 8s to 66s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 4.8s sooner and used 85 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 20 against 20 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

The device answered and the benchmark did not understand the answer. That is a bug here rather than a fault on the box, and the first such reply is kept so it can be chased without loading the model again.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 12.0x the wait, 0.55s at 72 tokens against 6.60s at 6,260. Reading got cheaper per token as the prompt grew, from 137 to 751 tok/s. Decode stayed near 50 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.05x the aggregate of one, and wall time grows in step with the callers, 7s to 53s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 0.7s and 33 tokens to the same question. The decode rate barely moved, 26 against 26 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 11.5x the wait, 0.60s at 72 tokens against 6.88s at 6,260. Reading got cheaper per token as the prompt grew, from 130 to 723 tok/s. Decode stayed near 23 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.00x the aggregate of one, and wall time grows in step with the callers, 6s to 49s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 2.0s and 86 tokens to the same question. The decode rate barely moved, 48 against 45 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 12.0x the wait, 0.57s at 72 tokens against 6.82s at 6,260. Reading got cheaper per token as the prompt grew, from 130 to 726 tok/s. Decode stayed near 55 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.13x the aggregate of one, and wall time grows in step with the callers, 7s to 49s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 1.8s and 67 tokens to the same question. The decode rate barely moved, 28 against 28 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 8.4x the wait, 1.30s at 72 tokens against 10.93s at 6,260. Reading got cheaper per token as the prompt grew, from 60 to 456 tok/s. Decode stayed near 10 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.24x the aggregate of one, and wall time grows in step with the callers, 19s to 131s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 19.7s sooner and used 424 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 20 against 19 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 19.2x the wait, 0.25s at 68 tokens against 4.88s at 6,256. Reading got cheaper per token as the prompt grew, from 319 to 976 tok/s. Decode stayed near 21 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.00x the aggregate of one, and wall time grows in step with the callers, 6s to 49s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 0.8s and 32 tokens to the same question. The decode rate barely moved, 51 against 49 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 17.8x the wait, 0.33s at 68 tokens against 5.85s at 6,256. Reading got cheaper per token as the prompt grew, from 219 to 810 tok/s. Decode stayed near 41 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.04x the aggregate of one, and wall time grows in step with the callers, 7s to 52s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 1.1s and 35 tokens to the same question. The decode rate barely moved, 26 against 26 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 21.2x the wait, 0.23s at 68 tokens against 4.83s at 6,256. Reading got cheaper per token as the prompt grew, from 321 to 976 tok/s. Decode stayed near 47 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.00x the aggregate of one, and wall time grows in step with the callers, 6s to 46s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.1s and 7 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

92x the prompt cost 21.5x the wait, 0.23s at 68 tokens against 5.00s at 6,256. Reading got cheaper per token as the prompt grew, from 313 to 942 tok/s. Decode stayed near 48 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.02x the aggregate of one, and wall time grows in step with the callers, 6s to 43s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking finished 1.3s sooner and used 36 fewer tokens than not thinking. That is not a measurement error: having reasoned, the model writes a much shorter answer, and the reasoning tokens are counted in the total. The decode rate barely moved, 30 against 30 tok/s.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

89x the prompt cost 20.5x the wait, 0.24s at 70 tokens against 4.89s at 6,258. Reading got cheaper per token as the prompt grew, from 328 to 966 tok/s. Decode stayed near 32 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.12x the aggregate of one, and wall time grows in step with the callers, 6s to 43s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.3s and 0 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

85x the prompt cost 16.1x the wait, 0.24s at 74 tokens against 3.82s at 6,262. Reading got cheaper per token as the prompt grew, from 326 to 1,239 tok/s. Decode stayed near 41 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.03x the aggregate of one, and wall time grows in step with the callers, 7s to 59s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 16.1s and 354 tokens to the same question. The decode rate barely moved, 22 against 22 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

95x the prompt cost 17.8x the wait, 0.25s at 66 tokens against 4.44s at 6,254. Reading got cheaper per token as the prompt grew, from 295 to 1,064 tok/s. Decode stayed near 34 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.02x the aggregate of one, and wall time grows in step with the callers, 5s to 43s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Thinking on and off came out the same, within 0.1s and 17 tokens. Either this model does not reason on this prompt, or it ignores the switch. Nothing here is a cost.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 19.5x the wait, 0.17s at 72 tokens against 3.32s at 6,260. Reading got cheaper per token as the prompt grew, from 496 to 1,427 tok/s. Decode stayed near 36 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.04x the aggregate of one, and wall time grows in step with the callers, 8s to 60s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 9.9s and 211 tokens to the same question. The decode rate barely moved, 21 against 21 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 12.1x the wait, 0.55s at 72 tokens against 6.67s at 6,260. Reading got cheaper per token as the prompt grew, from 136 to 744 tok/s. Decode stayed near 43 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.03x the aggregate of one, and wall time grows in step with the callers, 8s to 58s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time.

The same question asked twice. Hidden reasoning tokens are paid for in wall time whether or not anyone reads them.

Reasoning added 1.0s and 33 tokens to the same question. The decode rate barely moved, 23 against 23 tok/s, so that is the model thinking rather than the box slowing down.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.

What a long prompt costs to read, and what it does to the wait before the first word. Solid is time to first token on the left axis; dashed is decode rate on the right.

87x the prompt cost 11.3x the wait, 0.61s at 72 tokens against 6.88s at 6,260. Reading got cheaper per token as the prompt grew, from 127 to 722 tok/s. Decode stayed near 22 tok/s throughout, so the prompt is paid for once at the front and not again per word.

Every stream added together, against what each single caller saw, at each level.

8 streams deliver 1.08x the aggregate of one, and wall time grows in step with the callers, 7s to 7005s. The box is not sharing itself between requests, it is queueing them. Plan for one inference at a time. 6 request(s) failed and are not in these figures.

Sampled on a separate thread while the long generation ran, not before or after it. A reading taken once the request has returned always catches the box going idle and says nothing.