Volver
Grok BotMulti-AgentLLM RoutingWorkBuddy

I Put 3 Multi-Agent AI Products to the Test by Having Them Write the Same Research Paper. The Demos Were Great. The Results Weren't.

91% Útil9% No útil
0 Comentarios

I made 3 multi-agent AI products write the same research paper — great demos, broken results

xAI's Grok Bot got its own work email the other day. Cute. But here's what I actually wanted to know: strip away the slick demos, can a "team of AI bots" deliver real work? So I made three of them — Grok Bot, WorkBuddy, and Qoder CN — do the exact same research task.

The test

No softballs. One prompt, three products, same deadline. They had to write a survey on LLM routing with real citations, a 12-paper comparison table, and a separate fact-check pass. Here's the exact prompt:

1. Use your multi-agent collaboration features to write a Chinese research survey on "LLM routing," current as of October 15, 2026.

2. Assign different agents to literature search, method categorization, benchmark verification, and independent review — with one lead agent integrating everything.

3. Split the work however your product supports, but keep visible handoff and revision records. Don't fabricate collaboration that didn't happen.

4. Final deliverables:
   (a) a ~3,000–4,000 word survey explaining what LLM routing solves, how methods are categorized, and LLMRouter's contributions and limits
   (b) a comparison table of at least 12 original papers (method, routing signal, model pool, quality/cost/latency, applicability)
   (c) an "open questions" section
   (d) a short collaboration log — which agent caught what, who verified it, how it was fixed

5. Prefer reading full papers, author repos, and public code.

6. Link every key fact and number to an openable original source. Separate what the paper reports, what independent benchmarks show, and what's your own inference.

7. Mark anything you can't verify as "unverified" — don't fill gaps or invent citations.

8. Before finishing, have an agent that did NOT write the draft independently check citations, numbers, method classification, and scope.

The results

ProductTimeAgentsWhat it's best at
Grok Bot12:494 rolesfastest, most papers (19)
Qoder CN14:426 sub-agents + maincatching its own errors
WorkBuddy21:435 rolescleanest, most editable draft

What worked

Grok Bot is fast — 12:49, 19 papers, named bots handing work off. Looks great in a demo reel.

Qoder CN is the one that actually caught its own mistake: its reviewer flagged an unsourced "85% cost saving" claim from RouteLLM, then went and verified it in the repo. That's the only genuinely useful behavior in this entire test.

WorkBuddy is slow (21:43), but its draft is the only one an editor could actually salvage.

What didn't

Here's the part the marketing skips: all three shipped errors. Wrong citations. Inconsistent numbers. Mislabeled methods.

And here's the kicker — the corrections happened during the run. The bots talked, flagged issues, "fixed" them. Then the fixes never made it into the final files. You can watch five agents pat themselves on the back in the chat log, and still end up with a paper full of wrong numbers.

That's the whole story in one line: the demos are theater, the deliverable is broken.

Verdict

Speed is a distraction. The real fight is reliability — whether a fact-check survives the handoff and actually lands in the final document. In this test, it didn't.

Worth it? Not yet. These are impressive demos with sloppy deliverables. Use them for real research only if you're willing to verify every number yourself.

Comentarios0

Inicia sesión para comentar
  • Aún no hay comentarios.

Artículos relacionados