Skip to content
Bonrock
Back to articles
AI Strategy

What Klarna's First-Month AI Results Could Not Tell Us

Klarna's AI assistant matched human satisfaction scores in its first month, and a year later the company was hiring people back. What early pilot numbers cannot settle.

On 27 February 2024, a month after switching on its OpenAI-powered assistant, Klarna published its results. The assistant had handled 2.3 million conversations, two thirds of the company's customer service chats. Customers were finishing their errands in under 2 minutes, down from 11. Repeat inquiries had fallen 25%. Satisfaction scores were on par with human agents, and the company estimated a $40 million profit improvement for the year (Klarna).

That press release is worth reading closely, because Klarna measured more than cost. It reported speed, accuracy and customer satisfaction alongside the savings.

Fifteen months later, chief executive Sebastian Siemiatkowski told Bloomberg that the AI-first approach had been cheaper and had produced "lower quality" service, and that Klarna was recruiting human agents again (Entrepreneur). By June 2025 he was describing human support as something closer to a premium product, with the company's headcount down from about 5,500 to about 3,000 (TechCrunch).

Klarna has not published what went wrong, or how it evaluated the assistant after launch, so nobody outside the company can say why its view changed. The public sequence supports a narrower point: strong aggregate numbers from the first month did not settle the company's longer-term judgment about service quality. That is worth keeping in mind for anyone approving an AI pilot on the strength of early results.

What a month of averages can hide

An average satisfaction score can be excellent while a small group of conversations goes badly. In customer service, that group might be disputed charges, customers in financial difficulty, or questions the written policy does not quite cover. If those cases are uncommon, they barely move the average, yet they may weigh heavily in how customers judge a company over time.

We don't know whether that happened at Klarna. It is a possibility any evaluation should be designed to catch, because an aggregate is exactly the kind of number that hides it. The authors of the UltraChat paper wrote that "the journey from 0 to 60 is easy, whereas progressing from 60 to 100 becomes exceedingly challenging", a line Chip Huyen quotes as one of the first painful lessons of building AI products (Huyen). Aggregate early numbers can look the same for a system stuck at 60 and one on its way to 100.

The often-repeated MIT figure, that 95% of organizations get no return from generative AI, rests on 52 interviews and 153 survey responses, and its authors call the result directional (MIT NANDA). A failure rate across the market tells a manager little about the one project in front of them. The more useful question is what evidence would justify funding, changing or stopping it.

Evidence built from your own difficult cases

The practice we recommend is to collect real examples of the work before tuning the system, deliberately including the awkward ones, and write down what an acceptable outcome looks like for each. The system is measured against that set, and failures found later in production are added to it.

The effort can be large. Hamel Husain and Shreya Shankar, who have built evaluation systems for many AI products, report spending 60 to 80% of their development time on error analysis and evaluation (Husain and Shankar).

Morgan Stanley is a public example of evaluation before launch. According to its vendor OpenAI, the bank answered internal doubts about an advisor assistant by deciding "to conduct intensive evals for every proposed application", grading answers against its own experts before rolling use cases into production. OpenAI reports that "98% of Morgan Stanley advisors use OpenAI every day" (OpenAI). That is the vendor's account, and it shows one firm's sequence, without proving that evaluation is what separates success from failure everywhere.

A case set also turns some recurring arguments into measurements. Whether a step needs an agent can be tested by running the cases with a fixed script first; Anthropic's engineers advise starting with "the simplest solution possible" and adding complexity only when needed (Anthropic). Whether to build or buy can be tested by running a vendor product and an internal prototype against the same cases.

Shared ownership, with the business defining acceptable

We would split the work this way. The people who handle disputes and exceptions every day define what an acceptable outcome is, because they know which cases matter and what a good resolution looks like. Engineers investigate the failures, build the checks and keep them running. Each side needs the other: domain experts without engineers produce a list nobody can run, and engineers without domain experts risk testing what is easy to score.

The set needs maintenance. Policies change, products change and customers find new problems, so cases go stale and new ones need adding. Each model update or prompt change should be run against the current set before it reaches customers.

A practical starting point is to pull a sample of recent real cases, weighted toward the ones that were escalated, refunded or complained about, and have the people who resolved them write the correct handling next to each. A few dozen is often enough to reveal whether the team and the system understand the work; regulated data or slow adjudication may make the first batch take longer, which is itself worth knowing before the build begins. The size matters less than the mix. A set made only of easy cases will reproduce the comfortable averages of a first-month dashboard, and tell the budget owner nothing new.

Sources

Found this helpful? Pass it on.

Working on something like this?

We help organizations build AI systems that are secure, private and ready for real load.

Talk to us →