Beyond prototypes: shipping trustworthy AI

In May 2025 I gave the opening keynote at Nordic Testing Days in Tallinn. It's three stories about getting AI features into production: one from Veriff, one from NFTPort, one from Pactum.

The full video is on YouTube (about 28 minutes of talk, then half an hour of questions). Below are the slides with what I said about each one.

I opened by asking who in the room was not from Estonia. Most hands went up, which meant I had to explain the Estonian networking rules: never raise your hand, don't look people in the eye, and in the networking session only speak to people you already know. Then I told them three stories from building production AI features and products.

The first goes back to 2018, when I joined Veriff as their first machine learning engineer. Veriff is an Estonian unicorn now; what they do is identity verification, where you put in an ID card or passport and get back a decision about whether it's legitimate. When I arrived the AI part was not really there. Every decision was made by a human sitting in an office, and my job was to help the company grow without hiring hundreds more of them.

/

Veriff typically sits in a customer's onboarding flow. Someone signs up for Revolut (just an example, I don't actually know whether they're a Veriff customer), and part of signing up is verifying their identity document. If that fails, they can't use the product. So conversion is the metric that matters, and the thing killing it was users taking low-quality photos.

The loop went: user takes a photo, a human at Veriff reviews it manually, which can take an hour or 24 or in the worst case 48, and then if the photo isn't good enough they ask for a new one. If you've ever been on the receiving end of this you know how annoying it is. The edge of your document was half a millimetre out of frame, and you come back and do it again, three or four times.

What we wanted was the same loop in five seconds, with the feedback coming automatically instead of from a person.

When I joined, the way one of the engineers was building this struck me as very odd. He sat in the Tallinn office with a browser console open, writing pixel-counting algorithms and testing them directly against his own webcam. Which sounds reasonable until you look at his data, which was two photos of a white guy sitting calmly at a desk in front of a white wall. Nothing is happening in them. They aren't real usage data from actual customers.

Real data looks like this. Different genders, races and backgrounds, glasses with reflections in them, makeup, terrible lighting. And in one case there's no human in the photo at all, because the user has photographed their shoe or their curtains. None of that complexity was anywhere in the development loop that engineer was running.

He was fired about a month later. I don't think this was the only reason, but it reflected his approach, and I think it contributed to him leaving.

The better way is to take that variety and test against it. This is not rocket science. For each of those eight images you assign a label: is the image quality acceptable or not? You and I could label all eight in about fifty seconds. It's not a huge lift, which is part of why it's so strange that people skip it.

Then I asked the room a question. Same problem, detecting low-quality images. You could train a neural network from scratch. You could take a pre-trained network and train only a small model on top of its embeddings. Or you could take a classical computer vision algorithm like edge detection and hand-tune a pipeline around it. Which is best?

I got a show of hands for each. The correct answer was to not raise your hand at all.

It's a trick question, because the answer is whatever solves the examples well enough, and you should reach for the simplest thing that does. Which one that is can't be known in advance from the architecture; it's known from the examples.

So the loop is: list the examples, even just the eight we looked at. Define the expected output for each one, yes or no. Then measure the quality of whatever solution you currently have against that set, starting from a dummy that always answers "false", and iterate.

Every time I present this I feel a bit stupid, because it's such common sense. But I don't think it is common, because the default is different at each of those three steps. People start from a list of requirements rather than a list of examples. They write a spec with acceptance criteria rather than labelling outputs. And in development they focus on making all the tests pass rather than measuring quality against a dataset. Those are all reasonable practices for software where you can capture the input-output relationship in a couple of pages of rules. Most AI features are not that kind of software.

At this point someone in the audience is thinking: isn't this just TDD? It does share properties. You have a list of tests, the tests define the expected behaviour, and you keep improving the tests while you build the system.

The differences matter more. The tests are real examples pulled from production, not cases you wrote by hand, because the whole point is to capture complexity you wouldn't have thought of. It's mostly useful end-to-end rather than at unit level. You might have a thousand or a hundred thousand tests covering exactly the same input and output, each one sitting in a different part of the distribution. And the success metric isn't a pass rate. Accuracy is the simplest one, then precision, recall and F1 for binary classification, and it gets much harder once you're doing object detection or translation.

Learning number one: test against real examples, or you will be fired.

Second story. A couple of years ago some friends and I started NFTPort, which sold developer tools: APIs for creating NFTs and for using NFT data in your application. That photo is from peak NFT time, the summer of 2022, at NFT NYC, the industry's most important event of the year. Times Square was full of NFT billboards. I had never seen my industry own the streets somewhere like that. We had just raised a $26M Series A, we had around 30 people, and we were doing extremely well.

A couple of months later I read that headline. Mass adoption was the whole ballgame for us: if you make developer tools, someone has to use them and succeed with them. Some of the biggest social media platforms in the world were building on us, and we watched our contacts there get downsized and cut, week by week, emails going cold, no way to get anyone on the phone. Our biggest customer asked for a 95% discount, and they were right. That was the appropriate market price by then.

After a lot of difficult emotional processing, we decided to pivot. And what does a good founder do when they need to pivot? They fly to San Francisco.

The whole team had an AI background, so we went where the rawest, earliest action was. This slide is representative of the vibe at the time: someone central to the developer ecosystem (swyx) explaining how AI engineers would be a thousand times more productive.

This is mid-2023, and there was a project called AutoGPT. When I asked the room who had heard of it, plenty of hands went up, and I told them they'd see in a moment why they hadn't heard of it since.

At the time it was the fastest-growing GitHub project ever: 100,000 stars in three months. PyTorch took six or seven years to reach fewer.

AutoGPT was essentially the first AI agent, one of the first attempts at running an LLM in a loop with tools to do useful work. It had a complex prompt with separate sections for thoughts, reasoning, plan, criticism and speech, then picked an action. It had history and memory, web search, browser tools, a notepad. A huge number of bells and whistles, and we were excited about it.

As CTO I cloned the repo and started trying to improve on it. The first thing I did was build myself a test set, and one of the examples was finding the home address of Kaspar Peterson, my co-founder at the time. I knew it could be done from the public internet without huge effort, which made it a good test.

The set had about five examples in it. I started changing the codebase and running against those five, and what I found was that the code had a lot of moving parts, and as I removed them performance kept getting better. Faster and cheaper too, because it used fewer tokens. I was genuinely surprised. How can one not-especially-talented engineer improve on the thing with 100,000 stars that all of San Francisco is talking about?

I think I understand now. They didn't have a use case or a problem in mind, so they had no list of examples to improve against. They were adding features without knowing what the performance was. They raised a $12M seed round, which in Europe would be Series A scale, and a year and a half later there was still no product and the initiative appears to have died out.

Contrast that with OpenAI, at around 500 million monthly active users. There are many reasons for that, but a key one is that they have an extremely detailed evaluation suite. Part of it is a public repo you can go and read, which is unusual: every AI company invests heavily here, and almost none of it is visible from outside.

Learning number two: if you focus on the problem and define it well with examples, you won't over-engineer the solution.

Third story, and as you can tell from the photo, an intimate one.

I'm CTO at Pactum. We make software for Fortune 500 companies, specifically for the procurement department, which is where every dollar the company spends passes through. They're the ones managing all of that spending, they're our users inside those companies, and they have hundreds of different tools and a lot of very messy, unstructured data.

For us to automate anything there, or to run AI negotiations with their suppliers, which is the core of what we do, we have to understand every data format in the world. We can't assume anyone will send us neat JSON or CSV or XML. Images, screenshots, scanned paper documents, all of it.

The specific problem we wanted to solve in mid-2024 was finding contact details in any document. Invoices, contracts, quotes, emails, in image form or PDF or plain text, in any combination. Out of that the system has to produce the right person for Pactum to reach out to, so we can run a negotiation about the contract.

My hunch, which I'd had for a long time, was that you just put them into an LLM and it works. The intelligence in today's models is far more than enough for this. But a hunch isn't enough to put something on the roadmap and spend real time building it.

Three questions stood between the hunch and the decision. How effective would such a system be, on whatever the right quality metric is. How much effort would it take to get to that level. And could we avoid catastrophic mistakes, like extracting the wrong email from a PDF and sending sensitive business data to the wrong company.

We sat on this for a month or two. Eventually I'd had enough of talking about it, so I scheduled a hackathon. Not in the usual sense where you meet on a Friday evening, drink beer and think up business ideas, but one focused day with one of the best engineers in the company, where we already knew the problem and the exact questions we needed to answer.

We split the work. He wrote the scripts to pull in the data and built the first LLM-based extraction. My job was labelling data. This is the real spreadsheet; the black boxes are the cells I filled in by hand, looking at PDFs, text documents and images.

The engineer told me afterwards that he was surprised to see me focused entirely on the data and not at all on the code. I think it was the right call. The code part was, in a sense, simple. Working out what the correct answer is for each type of input, and assembling a dataset we could actually answer those three questions against, was the hard part. I'd do it again any day.

This gets more professional over time as a feature moves toward production. (Step 3 is garbled on the slide; it should read "Google Sheet + evaluation script".)

You start with nothing, no testing at all, maybe a proof of concept you poke at by hand. Then a couple of examples committed to the repo or sitting in a local folder. Then a Google Sheet like mine with a script that runs your candidate solution against it. Then a database, an API and a UI for labelling and reviewing. Many steps later you get to where we ended up at Veriff: production-like environments spun up on demand, parallel testing against hundreds of thousands of examples, structured output you can analyse in BI tools. About twenty people in operational teams supported that process, plus the infra engineers and data scientists.

You should still start from the Google Sheet. If you have nothing, taking the first step in one hour beats spending that time researching data labelling tools and testing frameworks.

The feature moved one of our main product success metrics by 30%. It kept a globally recognised brand from churning. And the third one is maybe the most important: succeeding at something narrow made us bolder. It raised our own sense of what we could pull off, and now we're thinking about things ten or a hundred times more complex in the same direction.

Learning number three: do things that don't scale.

I could have given essentially this same talk eight years ago. Nothing about the idea is new, and it isn't mine; I learned it at Starship, applied it at Veriff, and I'm applying it at Pactum. But there are two reasons it matters more now.

Most software is starting to include AI features, and you can't test those the way you tested everything else. That's an extra responsibility landing on all of our jobs.

And writing the solution is getting roughly ten times cheaper. Not a thousand times yet, but ten. With Cursor or Claude Code or Codex you can generate solutions automatically as long as the problem is well defined. Picture the time a technical person spends as a column: some slice of it is writing code, say half. That slice shrinks as the models and the tools improve, and everything else becomes a larger share of the total. Which means defining the problem well and validating whether the solution works becomes more and more of the job. Everyone in that room was becoming more important every month without doing anything at all, unless they were a very senior software engineer, in which case they might be in for a surprise.

To summarise: test against real examples, or you will be fired. Don't over-engineer the solution, or you'll miss out on a hundred-billion-dollar opportunity. And do things that don't scale, if you want to become the CTO.

Questions from the room

The Q&A ran almost as long as the talk. A few exchanges worth keeping.

What tools do you recommend for testing AI? I had a slide about five examples in a local folder, a Google Sheet and a script, and I meant it. Start thinking about tooling once you've done those and they've become too cumbersome. The problem is that end-to-end testing is deeply domain-specific. In the Pactum case I simplified for the talk: the input isn't really a document and the output isn't really an email, the input is an internal entity with multiple documents and multiple suppliers, and we want to evaluate the whole thing end to end. No vendor will hand you a perfect solution for that, which is why internal tooling has relatively more value there. For standard shapes, one image in and one label out, or text to text, there are tools. I used to maintain a list of every data annotation tool, findable by googling "awesome data annotation", though I haven't kept it up for a couple of years. Humanloop used to be good for text to text, Scale AI have expanded into this, LangSmith from the LangChain people has an open-source version, and for computer vision there's CVAT, free to self-host and a bit clunky. I couldn't give you one strong recommendation.

Will manual QA shrink or grow? You can feel how much intelligence a manual QA task requires. If you're not stretching your brain, if your contribution is reading a document and then moving a mouse and typing, that's relatively easy to automate, and vision and computer-use models are already good enough for a lot of it. Productising those models takes time. But someone still has to rein in the bots: decide what's important to test, look at what comes back and judge whether the task was completed correctly. If you've used an AI coding tool you know the job today is closer to being a kindergarten teacher, holding the model's hand so it stays on task and doesn't change things it shouldn't. There'll be less of that as systems improve, but I find it very hard to believe it changes drastically in the next two years, which in AI timelines is long term.

When do all the testers get fired? Look to your left, look to your right. I don't think there'll be an equal number of the same jobs with the same responsibilities in two years. But people and organisations move slowly, so getting to even half of you in the exact same role will take years. Then there's comparative advantage. I might be better than a twelve-year-old at literally everything, but they still have a job, because I have something more important to do than sell newspapers in the main square. Even if AI takes part of your job, there will be something left for you to do whether you like it or not. I'd expect salaries to go up, because you're achieving more in the same time with more powerful tools.

Where do you get data to annotate, and how do you know it's legitimate and diverse enough? Use your actual production usage data. That's the distribution your software will meet in operation, so it's the best thing to test and train on. Buying data is a bit sketchy: even where it's legal, the reputational risk is real, and if someone finds out you bought X amount of data from some platform, your clients won't like it. If you don't have production data yet, it's brainstorming time. You may be able to build a product or a feature that produces the data you need. Facebook were fairly transparent about this a few years ago with the ten-year challenge, where you uploaded a photo of yourself today and one from ten years ago, obviously to train their age models. You can try the same and make it feel more natural and less like data scraping than they did.

What skills do we need to get the most out of AI? Today the best predictor is curiosity and creativity. The standard response of "it made one mistake, throw it away" is a bad instinct; every technology in its early days has major flaws. Think about it the other way round: it's bad at most things, so what is it great at? Longer term I think it's general management skills, the ones that transfer from managing a person. Make the goal clear, give regular feedback on whether they're on track. Minus the encouragement and hugging, which AI needs less of. There's an underappreciated third thing. Part of the reason I'm on this stage is that I stopped doing public talks for three years, with two small kids and a startup to run, and put my effort into writing on my blog instead. Then models got good enough at writing that even if my quality is barely above the best of them, their quantity and personalisation is so much better that it's hard to compete. I'd like to see an AI come up here, make jokes and keep a smile on its face. Physical presence, personal relationships and being someone with a track record of doing things well matter more now, not less, as individual technical tasks get automated.

Do you say please and thank you to the models? Never, and I'm extremely brief. For a practical task I use as few words as possible, because typing takes time. If I want to know how to look after my monstera, my prompt is "monstera care 5 bullets". It's like learning to drop the unnecessary words from a Google search. My wife does the opposite and says please and thank you, and I think she appreciates it for different reasons. The AI companies have enough money to pay for the GPUs, so do whatever feels natural.