> ## Content Index
> Fetch the complete content index at: https://www.taivo.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# Hacky multimodality
- URL: https://www.taivo.ai/__hacky-multimodality/
- Published: 2023-05-04T17:30:32.000Z
- Updated: 2025-12-24T11:02:35.000Z
- Author: Taivo Pungas
- Tags: Systems, stream

**GPT*\-4 supports images as an optional input, according to OpenAI's press release. As far as I can tell, only one company has access. Which makes you wonder: how can you get multimodality support already today?

There are basically two ways for adding image support to an LLM:

1. Train a vision encoder that makes the image digestible for an LLM. This is what GPT-4 and the recently released [LLaVA](https://llava-vl.github.io/?ref=taivo.ai) do.
2. Hack support by converting images into text, and manipulating text using images. This can range from just OCR-ing the image and chucking the output into GPT, to multi-step workflows like [Grounded-Segment-Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything?ref=taivo.ai) or the paper (which I can't find now) that allowed an **Agent middleware* to use any HuggingFace model as a **Tool*.

Option (1) is the native and powerful one, whereas (2) is a limited hack. But a major benefit of the second approach is that you don't need access to the weights of the LLM (you can do it with API-only models like **ChatGPT* or **Anthropic AI*), nor do you need to train anything yourself. Which of course means we will see a lot of (2) in open-source and academic projects -- I expect much more juice to be pressed out of this category of fruit.