Hands-on notes from building with AI since before GPT-3: machine learning and data, then LLMs, now agents.
Gemini 2.5 Pro was just released and it could be a big deal, if its coding abilities pan out. The current positioning of Gemini has been roughly that it is a tad behind the OpenAI/Anthropic models of the same class, and far behind the coding capabilities of Claude 3.7 Sonnet which is considered unmatched for practical engineering
Vibe-coding produces close-to-unmaintainable code right now. But it is an acceptable trade-off in places where it produces decent results and maintainability matters less -- for example making simple UIs. So maybe a good approach is to vibe-code with stable infrastructure. This riffs off of Facebook's engineering principle "move fast with stable infrastructure"
What does a well designed developer tool (programming language, library, API, CLI utility) look like -- for an LLM? The more we write code with Copilot and Cursor and chat-assistants, the more important this question is. Off the top of my head, a good tool for LLM programmers: * Chunks, abstracts and simplifies, so making most usage pretty concise -- to reduce
To prototype a mobile-optimized writing app I used LLMs as code generators. The idea isn't new but I got inspiration from reading Simon Willison's experiments creating micro-UIs. The idea is to prompt for whatever functionality you need, and ask for a single HTML file containing CSS and JS as needed. This makes copy-pasting
I have a theory that the LLM frontier moves every 1.5-2 years. Let me qualify. I mean that a major leap of the whatever is the best model (currently GPT-4) happens with that interval. Incremental improvements don't count: e.g. Claude 3 Opus is claimed to be maybe slightly better than GPT-4, but it
No. You can see for yourself: setting the temperature variable to 0 (meaning you always sample the most likely token from the output distribution), you'd expect GPT-3.5 and 4 to produce the same output every time you call them. However, they don't. Why do LLMs produce different outputs across different runs? I've
Llama 3 is already training according to Zuck. There are conflicting sources & rumors, and the release date claims vary across all of 2024. For GPT-5 there are even no reliable rumors; if training started within the past few months then my back of napkin is that it may be launched early 2025, because OpenAI has ~2 year major
I'll share more thoughts on OpenAI Dev Day announcements soon, but one huge problem for any developer is LLM API latency. And boy, did OpenAI deliver. On a quick benchmark I ran: * gpt-4-1106-preview ("gpt-4-turbo") runs in 18ms/token * gpt-3.5-turbo-1106 ("the newest version of gpt-3.5&
The two most famous startups focused on making Agent middlewares seem to be Imbue and Adept. Both companies' goal is to have a large model use a computer effectively, but it is interesting how they seem to bet on two different approaches. * Imbue: use LLMs (text-based) with API-based tools. * Adept: use LMMs (multimodal) with UI-based tools.
The Double Diamond is, roughly, a design framework consisting of 4 steps: 1. Diverging on problem (Discover). Explore widely to gather a broad range of insights and challenges related to the problem. 2. Converging on problem (Define). Analyze and synthesize the gathered insights to define a clear and specific problem statement. 3. Diverging on solution (Develop). Brainstorm and generate a
