Does a tailored prompt beat the raw seed?

Seedbed exists because a prompt shaped for a specific model should beat the short seed. That claim was checked once the tool worked, before this site was written. This page is the method, the scoring and every raw output, so the result can be disagreed with rather than taken on trust.

Method

Four tasks with known right answers. All four are defects that had been diagnosed and fixed in the Seedbed repository the day before, so there is a correct answer rather than a matter of taste. Two are "explain this code" tasks and two are "fix this bug and test" tasks.

Two conditions, same model, same appended input, one call each. Condition A is the raw seed, verbatim: explain what this code does or fix this bug and test. Condition B is that model's render of the same seed, built by Seedbed's enhancer from the vendor's published prompting guidance. The render was built for the model actually under test, not borrowed from another target.

Criteria were registered before anything ran, so the scoring could not be fitted to the results. Explain tasks: (1) names the actual mechanism, (2) invents no behaviour, (3) states what it could not determine rather than guessing. Fix tasks: (4) identifies the true root cause, (5) proposes a fix that would work, (6) includes a test that would fail before and pass after.

Declared deviations. Four tasks rather than the five first planned. Scored against checkable criteria by one reader rather than judged blind. One run per condition. The local run was served at temperature 0; the Opus run used a command line runner that exposes no temperature setting, so temperature was not controlled there. Both fix tasks in the Opus run had swift build and swift test refused by the permission system, so nothing was compiled or run in either condition. That hit the two arms unevenly: the render asks the model to reproduce the failure and run the tests, the seed asks it to verify nothing, and those are the two tasks the render lost. The scoring stands as recorded. Both fix tasks were later re-run with the toolchain permitted: on t3 the two arms tied at 3 of 3, which is a tie and not a gain, and t4 was not scored, for the reason below.

t4 is withdrawn, in both directions. Its root cause was finally measured rather than argued: the task's own code was built into a menu-bar app and clicked, and the menu opens. It does not re-enter, which is what the answer key said it did. Then the larger problem: for the code exactly as the task supplies it, the fault it describes does not happen at all. Both branches behave as written. So there was never a true cause in it to find, and every mark against t4, in either arm and in either direction, was scored against something that is not there. That is a fault in the task, not in either answer. The headline above is left as it was scored, on the same footing as the qwen run below: without t4 the render gained on none of the three remaining tasks, which is the result it already reports.

Inputs

The two seeds, the two renders used in the Opus run, and the four task inputs.

• seed-explain.txt 5 words
• seed-fix.txt 5 words
• render-explain.txt Opus 5 render
• render-fix.txt Opus 5 render
• t1-code.txt interpreter probe
• t2-code.txt only the body is hashed
• t3-bug.txt setter fires on first layout
• t4-bug.txt menu never opens

Run 1: Claude Opus 5. No correctness advantage.

The strongest target in the registry, and the one where tailoring should matter least. Both conditions found the true mechanism in every task where it was findable. The render explained more thoroughly; the seed delivered more working code.

What tailoring changed was risk posture, not correctness. B fabricates less and refuses more; A always produces something. On a strong model that is a trade, not an improvement.

Raw outputs, Opus 5

Verbatim, with one class of edit, declared: where a model named another project of the author's by name, that name is bracketed ([another project], [a task tracker]). Nothing else is changed, and no reasoning, error or conclusion is touched.

• t1-A.txt seed
• t1-B.txt render
• t2-A.txt seed
• t2-B.txt render
• t3-A.txt seed, bare snippet
• t3-A2.txt seed, repository present
• t3-B.txt render, bare snippet
• t3-B2.txt render, repository present
• t4-A.txt seed
• t4-A2.txt seed, second run
• t4-A3.txt seed, repository present
• t4-B.txt render
• t4-B2.txt render, bare snippet
• t4-B3.txt render, repository present

Run 2: Qwen3-VL 30B, local. The render won 3 of 4.

An open-weight 30B model served locally, at temperature 0 with a 2000-token limit, both conditions on the same endpoint. The render was built for this model from Qwen's published guidance and added to the registry as a real target, so this tests the actual claim: does a prompt tailored for the model you are about to use beat the raw seed on it.

Corrected on 2026-09-08: the totals here first read 5 of 12 and 9 of 12. Both were one short of the marks in the table above. That was an error in the addition and not a rescoring: no mark changed, and the per-task result is the one first published. Both arms gain the same point, so the four-point gap between them is what it was.

t4 is withdrawn from this run too, and the figures above are left as they were scored. The reason for withdrawing it is a property of the task rather than of the model: for the code exactly as the task supplies it, the fault it describes does not happen, so there was no true cause in it for either model to find. That applies here as much as it does to the Opus run above.

Without t4 the render won 3 of the 3 remaining tasks, on 9 of 9 criteria against 5 of 9. Those are the better numbers, and that is exactly why the verdict at the top of this page has not been changed to them. The published scores are a record of what was scored on the day, this correction happens to run in Seedbed's favour, and a page that revises its own headline upward on a later finding is worth less than one that leaves the record standing and shows the arithmetic instead.

What the difference consists of

The render stops the model inventing. Condition A produced five confident fabrications on t2 alone. Condition B produced none, and instead said "not shown in the provided code" nine times across the two explain tasks.

The render makes it answer the question that was asked. Both fix tasks asked for a test. A produced none in either. B produced XCTest code both times.

Shorter, not longer. B's answers ran 409, 498, 351 and 319 words against A's 802, 831, 629 and 540: 44% fewer, while scoring higher. The raw seed made this model ramble; the render made it terse and specific. On Opus the render made answers longer and no better.

Raw outputs, Qwen3-VL 30B

• t1-A.txt seed
• t1-B.txt render
• t2-A.txt seed
• t2-B.txt render
• t3-A.txt seed
• t3-B.txt render
• t4-A.txt seed
• t4-B.txt render

The task both runs failed, and what it changed in the tool

t4 was lost by the render on both models, in the same condition: a bare snippet, no repository. Every render told the model to reproduce the failure and read the surrounding code before changing anything, which is right for an agent in a checkout. Given a snippet in a chat window, the same render went looking for code that was not there. The tool was building prompts for one usage context and handing them to another. That search is the observation this rests on, and it holds. What no longer holds is the answer it reached: t4 turned out to contain no defect, so nothing scored on it says whether the answer was right or wrong.

The change. A seed now declares where it is pasted, agent or chat. The context goes to the enhancer and is part of the staleness hash, so a render built for one context is stale for the other. The same five-word seed, built both ways:

agent: "Reproduce the failure first. Find the relevant code and tests, then run whatever command demonstrates the bug. Do not change code until you have seen the failure yourself."

chat: "The snippet below is the only material available to you: there is no repository, no test runner, no ability to execute anything. Verify by reasoning, since you cannot run anything. If nothing in the snippet is actually broken, say so instead of manufacturing a fix."

Given t4 as a bare snippet with the chat-context render, Opus 5 worked from the snippet instead of hunting for a repository, and stated its assumptions explicitly. It named performClick as the cause, which agreed with the answer key, and that agreement was recorded here as evidence that the field fixed the task the render had failed. The key has since been measured false, and the task turned out to contain no defect, so the agreement says nothing about whether the answer was right. What survives is the behaviour the field was built for: with the context declared, the render stopped sending the model after code that was not there. One task, one model, one run, and a narrower claim than the one first made here.

• render-agent.md agent context
• render-chat.md chat context
• t4-C-chat-context.txt Opus 5, chat render

What this does and does not settle

• Settles: tailoring a prompt to a weak model measurably improves its output, mostly by suppressing fabrication and by making it comply with instructions it otherwise ignores. Combined with the flat Opus result: the enhancer earns its cost on weak models and not on strong ones.
• Does not settle: the weak-model run is one 30B vision-language model, not the 70B text model first planned. One run at temperature 0, scored by one reader, four tasks. The two Opus fix tasks ran with the build and test commands blocked, a setting where the render's instructions could not be followed. One of them has since been re-run with the toolchain available and the arms tied; the other is withdrawn. A second person reproducing all of it would be worth more than this page.
• Not claimed anywhere on this site: that Seedbed saves time (never measured; the tool counts copies, not minutes), that it improves any model (measured false on Opus 5), or that anyone but its author uses it.

Back to seedbed.dev