🔍 Read the full analysis: How To Turn Database Grading Into RL Rewards With ThinkingBox on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, using executable checks of final database state as rewards for agent training. In a reported 50-step GRPO run, Qwen3.8-27B’s average success rate on a 507-task benchmark rose from 51.70% to 60.78%, while tasks passed in all 20 attempts increased from 38 to 107. The model solved seven fewer tasks at least once, and the report does not establish production performance or reproducibility.
Microsoft and Hugging Face say their ThinkingBox-Training stack is now available on GitHub, allowing AI agents to be trained with rewards based on whether their final database state matches an expected outcome, as detailed in the original analysis. In one reported 50-step run, Qwen3.8-27B improved its average success rate on a 507-task benchmark, although it solved slightly fewer tasks at least once.
The teams report that Qwen3.8-27B’s pass@1 rate—the share of individual attempts that succeeded—rose from 5,242 of 10,140 attempts (51.70%) before training to 6,163 attempts (60.78%) afterward. Each of the benchmark’s 507 tasks was evaluated 20 times. The report does not specify in the supplied material whether these attempts represent independent runs or how variability was measured.
The number of tasks that succeeded in all 20 recorded attempts, a measure the authors label Observed 20/20, increased from 38 tasks (7.50%) to 107 (21.10%). But pass@20—the share of tasks solved at least once—declined from 453 (89.35%) to 446 (87.97%). That is seven fewer tasks with any recorded success, so the reported results combine a higher average attempt success rate and more consistently passed tasks with a small reduction in task coverage.
ThinkingBox-Training uses practice tasks in ThinkingBox format, separate from the evaluation benchmark, and rewards an agent according to an executable check of its final database state. The check looks for expected records after the agent finishes rather than requiring one prescribed sequence of tool calls. The authors say they selected tasks using estimates of how difficult each was for the base model, then held the training pool fixed during the run.
Reliability Gains With Lower Coverage
Database-connected agents can change records that affect bookings, payments and other business operations. A fluent response or plausible sequence of tool calls does not prove that the requested change was made correctly—or that an unauthorized change was avoided. A check of the terminal database state gives training a direct way to reward the outcome that matters to the task.
The reported benchmark results suggest this approach may help make some outcomes more repeatable: more tasks passed in every recorded attempt after training. Yet the fall in pass@20 means the model did not expand its reach across the benchmark. Teams evaluating agents should look at both consistency and coverage rather than treating a higher average success rate as proof of broad improvement.
The figures are the authors’ reported results from a single described 50-step run, not evidence of reliability in live business systems. They provide a reason to test outcome-based rewards further, but do not settle whether the gains will hold across other models, workflows or deployment conditions.
Products Worth Considering

Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black

Bose TV Speaker (2nd Gen) Compact Soundbar with Bluetooth and HDMI Connectivity

LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDR True Black 400
As an affiliate, we earn on qualifying purchases.
From Database Grading to Training
ThinkingBox was developed as a way to evaluate agents based on the database records left behind at the end of a task. The described benchmark contains 507 tasks, with a backend, a set of MCP tools and a simulated user with private context. Executable checks compare the final database state with the expected result. According to the authors, this makes it possible to grade whether the required record is correct without demanding that an agent follow a particular human-authored tool-call path.
The training report describes extending those checks from evaluation into reinforcement learning. It says the team used a separate pool of ThinkingBox-format tasks and trained Qwen3.8-27B for 50 GRPO steps, then assessed a selected checkpoint on the benchmark. The authors describe the benchmark as out of distribution because its domains, databases and most MCP tools were not encountered during training. That is the authors’ characterization; the supplied material does not provide additional detail on the benchmark’s separation or validation.
For one hotel-booking task, the authors say a traveler requested a room downgrade requiring manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved the booking and payment state in all 20 attempts, while the authors report that all expected records matched in 19. This is a single task example, not a general measure of hotel-booking performance.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
Limits of the Reported Run
The supplied account describes one 50-step training run and does not report how results vary across random seeds, different training pools or longer runs. It also provides no full comparison with alternative reward designs or training methods, so it is not possible from these figures alone to isolate how much of the change comes specifically from database-state rewards.
The report does not establish performance in a live production setting, the cost of training, or whether the results generalize to workflows beyond the benchmark. It also does not explain why pass@20 fell by seven tasks. The hotel example illustrates one reported case, but cannot show how often similar approval-related errors occur across other tasks.
Reproduction and Broader Testing
With the stack available on GitHub, researchers and developers can inspect the implementation and attempt to reproduce the reported run. The source material does not give a schedule for further experiments or name a next release, so additional milestones have not been confirmed.
The next useful evidence would include repeated runs, tests across more models and task pools, comparisons against other reward approaches, and evaluations on workflows that reflect intended deployment conditions. For business use, those tests would need to assess both correct final records and harmful or unauthorized changes. Until such results are available, the reported gains should be read as benchmark findings rather than a guarantee of production reliability.
Key Questions
What is ThinkingBox-Training?
It is a Microsoft and Hugging Face training stack, reported as available on GitHub, that uses executable checks of an agent’s final database state as reinforcement-learning rewards.
What changed in the reported benchmark results?
After a 50-step GRPO run, Qwen3.8-27B’s pass@1 rose from 51.70% to 60.78%, and tasks passed in all 20 recorded attempts increased from 38 to 107. Pass@20 fell from 453 tasks to 446.
Why use database state as a reward?
A final-state check can score whether the records required by a task are correct, without requiring the agent to use one specific sequence of tools. It can also identify cases where an agent’s response sounds successful but the backend state is wrong.
Do the results show that the approach works in production?
No. The supplied report gives benchmark results from one described training run and does not provide production deployment results, repeated-run statistics or a full comparison with other training methods.
Primary source: Hugging Face · via ThorstenMeyerAI.com
