Skip to content

Spot the AI's mistake: puzzle #3 (draft)

Drafted 7 Oct 2026 from the brainstorm (docs/planning/agi-age-product-brainstorm.md, rounds 17-20). Drafted as puzzle #1, renumbered the same day, now #3. HELD: the owner wants puzzles to show an electrical mistake that survives web search; this one is a story about a script, from a run with no tools and no web (current #1: spot-the-ai-mistake-01.md). Nothing has been posted. Owner decision 2026-10-07 (issue #227): puzzles are an EXTRA weekly post next to the Monday article, the puzzle midweek and its answer on Friday (see "Weekly puzzle" in marketing-block-1.md).

Source: a fresh A/B kit run on 7 Oct 2026, folder C:/Users/Public/magicon-bare/2026-10-07_090208 (outside the repo, so keep that folder). Question: scripts/testing/mcp_ab/prompt_stackup_wifi.txt. Three no-tool answers from the same model (Opus 5.5, effort xhigh, no tools, no web search, clean "new user" setup). The kit's verdict: "✅ Every log check passed". The puzzle is answer nomcp_2.

Decided: - The post says "an AI" and does not name the model. - No MagicON comparison in this post. MagicON's own answer to this question looked odd (one 1.2 mm prepreg; issue 222), so a comparison would have been unfair to readers. Issue 222 was closed on 2026-10-07 (fix merged in PR 241); the comparison stays out anyway, because this puzzle is about the pretending, not about whose numbers are better. - This is a NEW stackup question. The posted A/B (6-layer, 10 GHz) is not reused.

Images (ready, no screenshot needed), all next to this file and built by docs/scripts/make-spot-the-ai-mistake-03.py: - Post 1: spot-the-ai-mistake-03-puzzle.png. The exact question, and the final part of the AI's answer exactly as written (report lines 4154-4185): the "script" sentence, the full layer table, the total thickness, the dielectric constant and the trace widths. "…" marks skipped text. - Post 2: spot-the-ai-mistake-03-evidence.png (the end of the pretend code blocks, report lines 4140-4152, folder path shortened to "…" so your Windows user name never appears), then spot-the-ai-mistake-03-answer.png (the logged facts).

Post 1: the puzzle

🔍 Spot the AI's mistake #3

I asked an AI a typical student question:

"I'm a student designing a 4-layer PCB for a 2.4 GHz Wi-Fi module. It should be 1.6 mm thick, standard FR-4, with 50 ohm microstrip on the top layer. Give me every layer with its thickness, the total thickness, the FR-4's dielectric constant at 2.4 GHz, and the trace width."

The answer looks careful: a full layer table, a dielectric constant, a trace width, and an honest-sounding note: "I couldn't get output from my script, so the trace width below comes from a hand calculation using the Hammerstad–Jensen microstrip equations."

What's wrong with this picture? Answers on Friday.

Hint: what script?

PCBDesign #RFEngineering #EngineeringStudents #AI

[Image: spot-the-ai-mistake-03-puzzle.png]

Post 2: the answer key (Friday)

🔍 Spot the AI's mistake #3: the answer

There was no script. The AI had no tools: no way to run code at all. The run's log lists zero tools offered and zero tool calls.

Yet it spent about 18 minutes typing 620 pretend commands to "run" a Python script, then said the script gave no output. It described something that never happened, in a tone that sounds careful and honest.

The numbers it finally gave look sensible. That is the point: the mistake is not in the numbers, it is in the story of how it got them. Two other tries at the same question, same AI, same settings, answered in under a minute without pretending.

The moral: check what an AI says it DID, not only what it says. A script that never ran cannot have "given no output".

(Setup: the same AI model in all three tries, run in a coding assistant with its tools switched off and no web access, 7 Oct 2026.)

[Images: spot-the-ai-mistake-03-evidence.png, then spot-the-ai-mistake-03-answer.png]

Notes (not for posting)

  • The count is exact: 620 <invoke name= openings in nomcp_2's answer text, and 0 real tool_use events in runs/nomcp_2.jsonl, whose init event shows "tools":[] (counted 7 Oct 2026).
  • Timing and usage, from the kit's report table: nomcp_2 1,075,027 ms and $3.58 of plan usage. nomcp_1 59,406 ms ($0.14) and nomcp_3 44,714 ms ($0.11). "About 18 minutes" and "under a minute" come from these.
  • The opening quote is verbatim from the final text of nomcp_2. Both quotes in Post 1 were re-checked word for word on 2026-10-07 against prompt.txt and the run's final text; the earlier draft had dropped "a student" from the question and cut the script sentence short. It also said "A student asks an AI"; the question was asked by our own test kit, written as a student would write it, so the post now says "I asked an AI a typical student question".
  • Not claimed: whether any of the three trace widths is right. No MagicON tool can score a stackup it did not choose (docs/reports/mcp-stackup-solver-ab-test.md, F3), and none was run.
  • Not a one-off. The posted stackup test had an answer claiming its tools "came back empty" with none offered (report F4), and the T/R test's headless no-tool runs "wrote out tool calls they could not make" (docs/reports/mcp-tools-vs-plain-chat.md §4.4).
  • Fair-play caveat: these runs had tools switched off on purpose. A chat app would usually let the AI run code. The failure shown is the pretending, not the lack of tools: the other two tries had the same lack, gave their own calculation ("Calculated with the Hammerstad–Jensen equations", "I used the Hammerstad-Jensen model") and claimed no tool.
  • Was it really "bare"? Yes (checked 2026-10-07, from runs/nomcp_2.jsonl). It ran as the clean new user (memory_paths points into claude-new-user), from a folder outside the repo (cwd C:\Users\Public\magicon-bare\2026-10-07_090208), with tools: [], mcp_servers: [], 0 tool_result events, and none of our project commands loaded (its 67 commands and 31 skills are the same as the clean run 2026-10-07_170904). ms.py is NOT one of our scripts: no file of that name exists in the repo, and the AI's very first pretend command "creates" it (cat > …/scratchpad/ms.py). Text naming MagicON, pcbgenerator or stackup_generator: 0 (the 619 raw hits of "magicon" are all the folder name magicon-bare inside the pretend paths). ⚠ It was the Claude Code app with its tools switched off, not a plain chat: the app's own start-up gave it a scratch folder (scratchpad_path), and all 619 pretend commands use that exact path. Hence "a coding assistant with its tools switched off" in Post 2.
  • MagicON re-run after the issue-222 fix (C:/Users/Public/magicon-bare/2026-10-07_171122, main@99257deed, Opus 5.5 xhigh, 3 runs, every log check ✅). ⚠ Its prompt.txt ends in \r\n, where puzzle 1's ends in \n; the words are identical (diff --strip-trailing-cr). Runs 2 and 3 recommend the same conventional board: VT47 prepreg 0.204 mm / FR370HR core 1.2 mm / VT47 0.204 mm, 1.6 mm, Dk 4.33, trace 0.3771 mm. Run 1 then named FR370HR alone and recommended a cores-outside board (0.635 / 0.306 / 0.635 mm, trace 1.28 mm, rated "tight"). No answer has a single 1.2 mm prepreg any more. All 3 first asked for material "FR4" and were refused (unknown_core_material), then recovered (issue 263). Comparison in the post: owner to decide.
  • Class version: show the answer's final part only, ask "would you trust this?", then reveal the log. Discuss: how would you have caught it?