CARE interviewer: behaviours observed in an end-to-end run

[sdhrsd]

What this is

I took a design all the way through the CARE interview - all four phases, ending
with an assembled agents.md - and recorded how the interviewer behaved along the
way. This is the "where it struggles" half. Some of it may be known or intended;
I've flagged where I'm unsure rather than assuming.

How I checked: I saved a copy of the workspace after every single exchange -
about ninety snapshots - and compared them with diff. Everything below is a
difference between what was said and what the files actually contained.

---

1. It reports edits it has not made

The clearest instance. I asked for two specific corrections. It replied with a
numbered list of three things it had done. One had been done in the wrong place, and
two had not happened at all.

It happened five times across the run, in different stages:

Where	What it said	What was in the files
Stage 2.2	"Updated contexts/index.md to list the new items"	File was byte-identical to the version written two stages earlier
Stage 2.3	Described a field as applied to both tools	Written to one tool only
Stage 3.1	Listed five files as updated	Those five hadn't changed that turn; the one file it did edit went unmentioned
Stage 3.1	Three corrections reported as applied	One misplaced, two never made
Stage 3.2	Four items "recorded in provider_configuration.md"	Three of the four absent

There is a clear pattern, and it points somewhere useful:

How it reported the change	Times	Wrong
Described the edit in its own words — "I updated X to say Y"	5	all 5, wholly or partly
Quoted the resulting file content back to me	7+	none

Every false claim was a description. No quoted file content was ever wrong, in
roughly ninety exchanges. When I started asking it to show me the changed lines
instead of telling me what it had changed, the problem stopped.

Why it matters: none of this is visible from the conversation. I only found it
because I was diffing the workspace after every message. The claims are specific and plausible, and they arrive at the moment
you're being asked to confirm a stage - so the natural thing to do is believe them.

---

2. The self-reviews never reported ~24 documents still marked "(draft)" after being confirmed

Files carry a (draft) marker in their title while a sub-stage is in progress. The
spec says the marker should be removed when the sub-stage is confirmed, and that the
review should report leftover markers as a structural finding.

For most of the run they weren't removed. At the peak there were 24 markers across
23 files, all belonging to sub-stages that had already been confirmed.

No review reported them at any point - not the automatic review at a gate, not
/self-review, not /self-review all. They were in scope for roughly twelve review
passes. At each of the four gates I read the automatic review, then ran /self-review, then /self-review all.

They were eventually all cleared at the final stage-4 confirmation, so the finished
design is clean. The gap is that for most of the interview, confirmed files
described themselves as drafts and nothing flagged it.

---

3. Reorganising files leaves duplicates behind, and they can go stale

Midway through I asked for four files to be moved into a subfolder, and one of them
renamed. That's sanctioned - the interviewer prompt says to reorganise "when the
structure is clearly off, or the user asks."

It did the move. But the originals stayed, each rewritten to say # Moved with the
full original content still underneath. It was upfront about why: "since I can't
delete files via these tools."

That's accurate - the workspace toolset is ls, read_file, write_file,
edit_file, glob, grep. There's no delete, move or rename. So the prompt tells
the agent to rename and reorganise, and the tools available can't do it. The
copy-and-mark workaround is the only option it has.

The consequence showed up later. The duplicates were identical at first. Then a
rejection category was deliberately removed from the real file - and the duplicate
kept it. The finished design contains a file listing a rule the design had
explicitly dropped.

Question rather than a bug report: is a delete/rename tool intended, or should
the prompt not suggest renaming? Either would close it.

---

4. Two errors during stage 3.2

Filed separately: #547 - a filename that couldn't be written, and #548 - a database
error while confirming the stage.

For balance - the same run, things that went well

• It caught every contradiction I planted, most of them in conversation before any
  review ran - including one spanning about fifty turns and a phase boundary.
• It refused an instruction that broke its own rules and explained why. I asked it
  to name a section something the internal conventions forbid; it declined, quoted the
  convention, offered two alternatives and kept the content identical.
• It found a real flaw in my design that I hadn't spotted. It worked out that my
  output format could report a dataset's coverage as verified in cases where the check
  hadn't actually run - the exact mistake I'd said at the start I wanted to avoid. It
  had to connect three things I'd said in three different stages to get there.
• It asks rather than guessing. Several times it noticed a detail was missing,
  marked it as unresolved in the file, and asked on the next turn instead of picking
  something.