AI Agents ACT NOW OPPORTUNITY

Microsoft's free test kit for choosing the AI model behind an agent

ShiniFrontiers · Source: Microsoft

Section 1 of 6 FREE PREVIEW

What problem does it solve?

What it is: ThinkingBox is Microsoft's free test environment for AI agents, with 507 business scenarios and published results for 18 models.

Who it's for: companies choosing the AI model behind an agent. Beyond Microsoft's 18, a team can connect any other model and score it on the same scenarios. It shows which model gets the work right every time, and what each reliable result costs.

The catch: the scenarios are Microsoft's made-up ones, not your own workflows, and running it yourself needs a developer.

A. Choosing the AI model — for companies choosing the model behind an agent

Problem: An agent runs on one AI model, chosen before it goes live. That choice is usually made from rankings or price per answer, which reflect a single attempt, while a live agent repeats the same work thousands of times.

What it does: Start from Microsoft's published results, then run your shortlist, or any model they do not cover, through the same scenarios. The tables in section 2 show what that comparison reveals.

B. Testing agents properly — for teams that build agents

Problem: Most teams judge an agent by its final reply, yet an agent can report "Done, your ticket is closed" while leaving the record wrong.

What it does: ThinkingBox checks the records the agent leaves, not its reply. Running the scenarios shows a team which successes were false, and whether a change to its agent actually helps.

See the difference

The situation	Today	With ThinkingBox
Choosing the AI modelChoosing the model an agent will run on	Pick GPT-5.6 Sol, the cheapest per correct answer ($0.127)	Sol is right every time in only 82 of 507 scenarios; Claude Opus 5.5 in 241, at $7.80 each against $9.76
Choosing the AI modelA new model appears that no published comparison covers	Rely on the vendor's own claims	Connect it and score it on the same 507 scenarios
Testing agents properlyA customer's parcel is stuck, and the agent marks her ticket as solved	Every step looks correct, so the test passes	Fails: the delivery problem is still open, so the ticket should be on hold

The parcel case is Microsoft's own example, from one of the 507 scenarios.

What it saves: Money spent on a model that is right once but not every time, and mistakes found by customers instead of tests.
What it costs: The software is free. Each test pays an AI company for the model's work; Microsoft priced one run of all 507 scenarios at $43.49 for GPT-5.4.
How ready: Working in the real world

Version 1.0. The scenarios are Microsoft's made-up ones, not your own workflows, and running it yourself needs a developer.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

Join free to read it all →

Section 2 of 6 MEMBERS

How does it work?

Join free to read this section

Microsoft's free test kit for choosing the AI model behind an agent — how it fits together.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

The big picture

Think of a new bank clerk on trial. You do not take their word that an account was updated: you open the account and check, and you watch them do the same job 20 times before you trust them. ThinkingBox runs that trial for an AI agent, scenario by scenario, on a practice copy of a company's system.

How one scenario is tested

A scenario goes from the top to a pass or fail at the bottom. Green is new.

The scenario. A practice scenario: practice records, a customer's request, the agent's tools, the company's rules, and the right end result. A pretend customer: an AI that gives details, such as a booking number, only when asked. 507 made-up scenarios from shops, car insurance, travel, an online-only bank and consulting.

The AI. A settings file: names the AI models for the agent, the pretend customer and the checker; one model can play all three. Any model that can use tools can be connected, so new models are scored on the same scenarios. The agent never sees the right answers.

The practice system. A fresh practice system: the agent's tools, connected through MCP, on clean records for every try. A search service: Typesense 30.1, which the team switches on with Docker. Every try starts from the same clean records, so 20 tries are 20 fair tests.

Checking. The checker: lists every change the agent made and compares it with the right end result. 477 of 507 scenarios are checked on the records alone; 30 also check the reply.

The result. A results file and a problems file: pass or fail for each try; technical faults go to the problems file. Technical faults are run again, not counted against the model. Exact versions are recorded, so others can check a result.

ThinkingBox: technical view

What a company learns from comparing models

Ranked by price per correct answer, GPT-5.6 Sol is the cheapest of these three. Ranked by scenarios right every time, it comes last, and each of those costs more. An AI model that costs less for one successful attempt may cost more when you need it to succeed every time.

Model	Price per correct answer	Scenarios right every time (of 507)	Cost per scenario right every time
GPT-5.6 Sol	$0.127	82	$9.76
GPT-5.4	$0.131	128	$6.80
Claude Opus 5.5	$0.276	241	$7.80

Model	Right on one try	Scenarios right every time (of 507)	Cost per scenario right every time
GPT-5.4	65.36%	128	$6.80
GPT-6 Astra	58.31%	231	$7.45
Claude Opus 5.5	67.16%	241	$7.80
GPT-5.6 Sol	61.91%	82	$9.76
Claude Opus 5	66.50%	241	$13.30
Claude Sonnet 4.6	59.19%	102	$15.56
GPT-5.2	46.28%	44	$19.95
Kimi-K3	57.37%	68	$20.68
Qwen3.8-27B	51.70%	38	$24.36

What a team learns from the failures

Most failures can be fixed without changing the model. About two in three failed tries (67.24%) looked fine from outside: the agent finished, changed a record and reported no problem. Four in five failures came from tool trouble, not poor reasoning.

Main cause	Share of failures	In everyday terms
Tool trouble	79.9%	A tool gives an error or finds nothing, and the agent does not try another way
Wrong record update	10.3%	The right form, filled in with the wrong value
Customer left without an answer	7.0%	The paperwork is done, but the question was never answered
Nothing changed	2.9%	The agent talks but never acts

In failed tries that looked fine, the checker found	Share
A wrong value in a record	77.61%
An extra change nobody asked for	43.30%
A change that should have happened but did not	25.36%

Part A: how the cost per scenario is worked out

Who	What happens	In everyday terms
1. ThinkingBox	Runs the model through all 507 scenarios, 20 times, on clean records each time	The same exam, sat 20 times
2. The checker	Marks a scenario right every time only if all 20 tries pass	Only 20 out of 20 counts
3. Microsoft	Prices the model's usage at list prices	Adds up the bill
4. Microsoft	Divides the bill by the scenarios right every time	What each dependable result costs

Part B: how one try is checked

Who	What happens	In everyday terms
1. The pretend customer	Asks for help, and gives private details only when asked	A caller answering the clerk's questions
2. The agent	Follows the company's rules and changes records with its tools	The clerk works the case
3. The checker	Compares the records with the right end result	An auditor checks the files

Join free → I have an account

Section 3 of 6 MEMBERS

How to set it up, and what it costs

Join free to read this section

Microsoft's free test kit for choosing the AI model behind an agent — steps and cost.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

Before you start: is it a fit?

	ThinkingBox
Good fit if	The business uses, or plans, an AI agent that changes records: tickets, refunds, bookings, claims or accounts
	The team is choosing, or switching, the AI model behind that agent
	The team tests its agent by reading what it says, not by checking the records
Who acts on it	Head of operations or CTO for the model choice; an engineering lead to run the scenarios
Not yet if	The agent only answers questions and never changes a record
	The team needs to test its own workflows; the source does not explain how to add them

Part A: Choose a model

Shortlist two or three models from Microsoft's published results, on scenarios right every time and their cost (the head of operations) Use the tables in section 2, and weigh the kind of business closest to yours.
Score any model missing from those results, or confirm the shortlist (the engineering lead) Connect each model in the settings file; it must be able to use tools. Part B shows how to run it.
Compare the results, and say how you counted (the engineering lead) Choose how many tries the job deserves, and say whether you counted right at least once or right every time.

Part B: Run the scenarios

You need	Why
A developer	To install it and switch on three services
Linux, or Windows with WSL, plus Python 3.11 or newer, uv and Docker	The setup Microsoft tested
Access to the models to score, plus models for the pretend customer and checker	Any model that can use tools, through an OpenAI-style interface or a short custom connector; one model can play all three
A settings file, thinkingbox.yaml	Written by following docs/llm_endpoint_config.md in the microsoft/thinkingbox repository

Download OpenEnv and the ThinkingBox add-on (the developer)
Download the 507 scenarios, at the fixed release (the developer)
Install the ThinkingBox command-line tool (the developer)
Switch on the search service, in a second terminal window (the developer) Expect: the health check answers and the command finishes.
Switch on the agent's tools, in a third terminal window (the developer)
Switch on the OpenEnv server, in the first window (the developer)
Check it is ready (the developer) This check cannot see the search service or every AI model connection; check those separately. Expect: a ready answer; 503 means the scenarios, the settings or the tools are not ready yet.
Run one shop scenario (the developer) Expect: pass or fail in the results file (results.jsonl); technical faults in the problems file (errors.jsonl).
Study a failed try that looked fine (the engineering lead) Find a try that finished with no error but failed, and see what it changed in the records. It shows what your own tests miss.

Part C: Apply the lessons to your agent

Check the records before saving a change, not the agent's summary (the engineering team)
Sort tool errors so the agent retries only those a retry can fix (the engineering team)
Give the agent only the tools the job needs (the engineering team)
Make a person approve changes that cannot easily be undone (the head of operations)

These are Microsoft's suggestions; it has not measured how much each one helps.

What it costs

Item	Cost
The software	Free and open source (MIT; the scenarios under CDLA-Permissive-2.0; the OpenEnv add-on under BSD-3-Clause)
One run of all 507 scenarios	Microsoft's examples: $43.49 for GPT-5.4, $86.03 for GPT-6 Astra
All 20 runs	$800.00 to $3,206.00 for the nine models in section 2; the source does not say whether the pretend customer and checker are included
People	Developer time to set it up and run it

Join free → I have an account

	What changes	What it can mean for the business
Choosing a model	Models are compared on scenarios right every time, and what each costs	Less spent on a model that costs more but is no more reliable
Testing	The agent is judged on the records it leaves	Mistakes caught in testing, not by customers
Fixing	Every failure gets a main cause	Engineers fix tool handling first, where most failures are

Limit	Why it matters	What to do
Made-up scenarios, Microsoft's own results	All 507 scenarios are invented, and the blog reports no outside check.	Run your shortlist yourself before relying on the table.
Five kinds of business	Results change a lot between them.	Weigh the kind closest to yours.
Costs are estimates	List prices on 20 Sep 2026, not real bills; the tool itself does not report cost.	Use them to compare models, not to budget.
Setup takes effort	Three services, Docker and three AI models.	Check each service is healthy before testing.
Fixes not measured	Microsoft has not measured its suggested fixes.	Test each one with ThinkingBox.

Microsoft's free test kit for choosing the AI model behind an agent

What problem does it solve?

How does it work?

The big picture

How one scenario is tested

What a company learns from comparing models

What a team learns from the failures

Part A: how the cost per scenario is worked out

Part B: how one try is checked

How to set it up, and what it costs

Before you start: is it a fit?

Part A: Choose a model

Part B: Run the scenarios

Part C: Apply the lessons to your agent

What it costs

What do you gain?

What are the limits?

What this project can't do yet

What it does not do

Sources