> ## Content Index
> Fetch the complete content index at: https://www.taivo.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# GPT-3.5 and GPT-4 response times
- URL: https://www.taivo.ai/__gpt-3-5-and-gpt-4-response-times/
- Published: 2023-05-11T12:26:32.000Z
- Updated: 2026-08-04T11:24:49.000Z
- Author: Taivo Pungas
- Tags: Building with AI, stream

Some of the *LLM* apps we've been experimenting with have been extremely slow, so we asked ourselves: what do *GPT* APIs' response times depend on?

It turns out that response time mostly depends on the *number of output tokens generated by the model*. Why? Because [LLM latency is linear in output token count](https://taivo.ai/stream/%5F%5Fllm-latency-is-linear-in-output-token-count?ref=taivo.ai). But the theory isn't crucial here: we can directly measure how much latency each token adds by calling LLMs with different `max_tokens` values.

So that's exactly what I did. I tested *GPT*\-3.5 and *GPT*\-4 on both Azure and OpenAI. For comparison to *Open LLM*s, I also tested *Llama*\-2 models running on *Anyscale* Endpoints.

Here are the results. Each pane is a model/deployment combination; the horizontal axis is output token count, vertical axis is total response time, each dot is one measurement, and the trend line is the best linear fit to these measurements.

![Response time rising linearly with output tokens; Azure adds 25-28ms per token, OpenAI 35-94ms](https://storage.ghost.io/c/4a/28/4a28c12b-061d-4fef-9be7-3d9f25838589/content/images/2023/09/llm_response_time_linear.png)

- OpenAI *GPT*\-3.5: **35ms** per generated token.
- Azure *GPT*\-3.5: **28ms** per generated token.
- OpenAI *GPT*\-4: **94ms** per generated token.
- Anyscale *Llama*\-2-7B: **19ms** per generated token.
- Anyscale *Llama*\-2-70B: **46ms** per generated token.

**Azure is 20% faster for the exact same *GPT*\-3.5 model!** And within the *OpenAI* API, *GPT*\-4 is almost three times slower than *GPT*\-3.5\. The smallest *Llama*\-2 model is only 2.5 times faster than the biggest one, even though the difference in parameters is 10x.

You can use these values to estimate the response time to any call, as long as you know how large the output will be. For a request to Azure GPT-3.5 with 600 output tokens, the latency will be roughly `28ms/token x 600 tokens = 16.8 seconds`.

Or if you want to stay under a particular response time limit, you can figure out your output token budget. If you are using OpenAI *GPT*\-4 and want responses to always be under 5 seconds, you need to make sure outputs stay under `5 seconds / 0.094 sec/token = 53.2 tokens`. (Optionally, enforce this via the `max_tokens` parameter.)

These numbers are bound to change over time as providers optimize their models and deployments. For example, between May and August OpenAI was able to make their GPT-4 more than twice as fast per token! However, I expect these coefficients to change over the course of months, not days, so developers can consider them roughly constant within any given month.

Latency may also vary with total load and other factors -- it's hard to tell without knowing details about the deployments. However, the key thing to remember is this: **to make GPT API responses faster, generate as few tokens as possible**.

If you want to dive deeper, see my post on [how to make your GPT API calls faster](https://taivo.ai/stream/%5F%5Fmaking-gpt-api-responses-faster?ref=taivo.ai).

A caveat: these numbers will change over time as providers optimize their models and deployments -- e.g. between May and August OpenAI was able to make their GPT-4 more than twice as fast per token! However, I expect them to change over the course of months, not days, so from a user perspective you can consider them constant, and maybe review your model choice every now and then.

### Experiment details

Here's how I ran my experiments.

- Made an API call for each value of `max_tokens`, per model and provider. (Three repetitions for each combination to understand variance).
- Fit a linear regression model to the outputs, including intercept.
- Reported the coefficient as "latency per token".

And some more boring details:

- Experiments were run on 09 August 2023.
- Network: 500Mbit in Estonia (at these scales network latency is anyway a very small part of the wait).
- For OpenAI, I used my NFTPort account.
- For Azure, I used US-East endpoints.
- For Anyscale, I used my personal account.