> ## Content Index
> Fetch the complete content index at: https://www.taivo.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# LLM latency is linear in output token count
- URL: https://www.taivo.ai/__llm-latency-is-linear-in-output-token-count/
- Published: 2023-09-21T10:34:51.000Z
- Updated: 2026-01-17T19:38:30.000Z
- Author: Taivo Pungas
- Tags: Building with AI, stream

All top LLMs, including all **GPT*\-family and **Llama*\-family models, generate predictions one token at a time. It's inherent to the architecture, and applies to models running behind an API as well as local or self-deployed models.

Armed with this knowledge, we can make a very accurate model of what the LLM's response time will be in any given situation.

The basic formula is this: `T = const. + k N`, where:

- `T` is the total response time you see to a query sent to an LLM.
- `const` depends on a whole host of things outside your control: DNS lookups, proxies, queueing, and *input* token processing.
- `N` is the number of *output* tokens generated by the model during your request.
- `k` is a measure of how long it takes to generate one token. It's a function of the model specifics on the one hand: its size, quantization, any optimizations that have been applied, etc. On the other hand, it also depends on the execution environment: hardware, interconnect, potentially even CUDA drivers, etc. From a developer's perspective, though, `k` is a relatively stable value and out of your control (unless you're in charge of your own LLM deployment).

For example, using **GPT*\-4 on **OpenAI* from my laptop and office network, the constant is around 1 second. The `k N` component is around 47 seconds. In general, unless you're making very small queries (outputting only 20-30 words or less) or using an extremely small model (smaller than **Llama*\-7B), the `k N` component will dominate the response time.

You can measure the value of `k` for any given deployment, and that's exactly what I have done in [GPT-3.5 and GPT-4 response times](https://taivo.ai/stream/%5F%5Fgpt-3-5-and-gpt-4-response-times?ref=taivo.ai).

Linearity is to me unintuitive in this context (often there are diminishing returns to these sorts of optimizations). So the important thing to remember is: it doesn't matter how many tokens you're generating now; a 2x reduction in token count will get you a 2x reduction in latency. It's scale-independent. If you're looking for concrete tips, see my post about [how to make LLM responses faster](https://taivo.ai/stream/%5F%5Fmaking-gpt-api-responses-faster?ref=taivo.ai).

You might also wonder (I did!) why the response time is not linear in input token count. The answer is that input token processing is embarrassingly parallel: the embedding for each token is independent of other tokens in the input -- because they are all known in advance. For output, however, generating the N-plus-1-th token needs to look at the N-th token, so every token will necessarily add some latency.