Skip to content

Design ChatGPT web support for EGM (1 of 2) - #435

Open
n3wth wants to merge 6 commits into
mainfrom
docs/egm-chat-design
Open

n3wth wants to merge 6 commits into
mainfrom
docs/egm-chat-design

Conversation

@n3wth

@n3wth n3wth commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary of Key Changes

Refs #414. Part 1 of 2 in stack #472: #435 design → #464 implementation.

Design ChatGPT web support in the existing EGM plugin and listing, using one OpenAI archive for web and desktop.

  • Define egm-in-chat for document brainstorming, drafting, and explicitly same-context review.
  • Check actual chat capabilities before repository work or independent review.
  • Package the document skill from openai/skills/, preserving shared coding-harness discovery. OpenAI has nine skills; Claude and repository installs have eight, including engineering Lite from feat(skills): implement EGM Lite rapid engineering protocol (Stage 2) #438.
  • Define host verification and publication gates; preserve recorded evaluations with their limits.

Goals and Non-Goals are in docs/designs/egm_chat.md. The shared /egm router has been removed from this proposal and extracted to stack #476: #474 design → #475 implementation, for #327. It is not a dependency of this stack.

Review this design before #464. The implementation prepares v0.2.0 and the checklist for updating the existing OpenAI listing. Merging the stack does not publish the plugin. Individual PR auto-merge is disabled for native stack membership.

Contribution Type

Track: Feature / Architectural Change -- Stage 1: Design Doc PR

Verification

  • Design links and Markdown tables passed in the updated implementation's unit suite.
  • egm-review and egm-resolve: not run.
  • Recorded model evaluations: not rerun; they do not validate the current release candidate or host capability checks.
  • Pending: maintainer scope approval. Implementation and launch evidence remain in Add ChatGPT web support to EGM (2 of 2) #464.

Human Accountability Affirmation

@n3wth n3wth added the docs Improvements or additions to documentation label Nov 13, 2026
@n3wth
n3wth enabled stacked auto-merge (squash) November 13, 2026 08:07

@egm-triage-bot egm-triage-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.

@egm-triage-bot egm-triage-bot added the triage: passed Automated Stage-1 PR triage found zero blocking or major issues label Nov 13, 2026
@egm-triage-bot
egm-triage-bot requested review from a team and seanstappas and removed request for a team November 13, 2026 08:07
@n3wth
n3wth force-pushed the docs/egm-chat-design branch from f02f8b1 to eebeb8a Compare November 17, 2026 02:09
@n3wth n3wth changed the title Propose EGM Chat skill for chat hosts Propose EGM in Chat skill for chat hosts Nov 17, 2026

@egm-triage-bot egm-triage-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.

@n3wth n3wth mentioned this pull request Nov 17, 2026
1 task done
@n3wth
n3wth force-pushed the docs/egm-chat-design branch from eebeb8a to e97e967 Compare November 17, 2026 02:29

@egm-triage-bot egm-triage-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.

@n3wth n3wth changed the title Propose EGM in Chat skill for chat hosts Propose one EGM plugin for ChatGPT web and desktop Nov 17, 2026

@egm-triage-bot egm-triage-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.

@n3wth
n3wth disabled stacked auto-merge November 18, 2026 09:10
@n3wth
n3wth added this pull request to stack #472 November 18, 2026 09:10
@n3wth n3wth changed the title Propose one EGM plugin for ChatGPT web and desktop Design one EGM plugin for ChatGPT web and desktop Nov 18, 2026
Refresh the design branch with the merged EGM Lite implementation and
document nine OpenAI skills and eight canonical skills in v0.2.0.

@egm-triage-bot egm-triage-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.

@n3wth n3wth mentioned this pull request Nov 18, 2026
1 task done

@egm-triage-bot egm-triage-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.

@n3wth n3wth mentioned this pull request Nov 18, 2026
1 task done
@n3wth n3wth changed the title Design one EGM plugin for ChatGPT web and desktop Design ChatGPT web support for EGM (1 of 2) Nov 18, 2026
Comment thread docs/designs/egm_chat.md

Store the document skill in `openai/skills/egm-in-chat/`. The OpenAI packer places its runtime files at `skills/egm-in-chat/` alongside the eight canonical skills, including `egm-lite`, using the same path-safety and reproducibility checks. The Claude build and repository-based coding-harness skill discovery retain those eight canonical skills.

Remove `requires_local_executor` from both OpenAI manifest locations. Update the listing description to distinguish document work from independent review. This prepares a candidate; it does not publish an update to the current desktop-only listing.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing requires_local_executor reaches further than "prepares a candidate". The Codex catalog installs from the repo root (.agents/plugins/marketplace.json has "source": "./"), and workspace import uses the same path, so those installs lose the flag as soon as #464 merges. docs/codex.md says the flag is there because bootstrap needs shell, and tests/unit/test_harnesses.py asserts it is set.

Can you state this as the main decision, list the affected install paths, and add a desktop egm-bootstrap shell check to the gate at L218? If desktop loses execution, stripping the flag at pack time for the OpenAI target could be the fallback.

Comment thread docs/designs/egm_chat.md

Remove `requires_local_executor` from both OpenAI manifest locations. Update the listing description to distinguish document work from independent review. This prepares a candidate; it does not publish an update to the current desktop-only listing.

All full skills already read `docs/codex.md` before starting on OpenAI hosts. That document adds a capability check before any workflow intake: readable bundled resources and writable artifacts for full EGM; verified fresh-context native subagents for any Goldfish stage; repository and command access for repository operations. If a requirement is absent, stop that workflow, name the missing capability, and offer `egm-in-chat` when it is installed. Do not silently replace an independent review with same-context critique.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads as if the check already exists, but #464 adds it. Every full skill loads docs/codex.md only "When running in Codex or ChatGPT Work", so ChatGPT Chat (where the pilot ran, L144) may never read it. Please phrase this as a proposed change and widen that trigger to all OpenAI hosts. The stop rule against same-context Goldfish review is already inline in each full skill, which is worth saying here.

This change also reaches Codex CLI and IDE users, so Required Tests should include a Codex CLI run of brainstorm, draft, and review.

Comment thread docs/designs/egm_chat.md

- **Given:** The same OpenAI package is installed in a chat with readable bundled files, writable artifacts, and verified fresh-context subagents.
- **When:** The user requests independent review.
- **Then:** EGM routes to `egm-review` and retains its isolation checks. A document-only request can still use `egm-in-chat` in that chat.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All full skills set disable-model-invocation: true, so "EGM routes to egm-review" can only mean telling the user which skill to select. Can you say that here so the design does not imply an automatic handoff?

Comment thread docs/designs/egm_chat.md
- Unit tests check that the existing OpenAI archive has the same `egm` identity, contains all nine skills and their runtime assets, omits `requires_local_executor` in both manifests, and is reproducible. The Claude archive has eight skills. Both archives must include `egm-lite`; only OpenAI includes `egm-in-chat`. Missing or symlinked document-skill files fail before output.
- `ruff check .` and `python -m pytest tests/unit` pass.
- Offline package validation passes for the combined OpenAI package.
- Before merge or publication, install the exact same archive in clean ChatGPT web and desktop chats. Record archive checksum, package version, host version or date, installation result, skill discovery, and observed capabilities separately for each surface.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and at L224, "before merge" also reads as blocking this design PR, which would make it wait on #464. Can you split the gates: what blocks #435, what blocks #464, and what blocks publication?

Comment thread docs/designs/egm_chat.md
The skill tied the plain model on five of six pairs and lost one.
No reply invented facts or claimed independent review.
ChatGPT Work listed all packaged skills; the UI did not show whether the skill file was read.
The pilot must be repeated on version 0.3.0 before Stage 2 merges.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gate needs a pass bar. The only on-host result so far is 5 ties and 1 loss, so as written it cannot fail. Something like "no losses against the plain model and no independence claims across N blind pairs" would work.

Comment thread docs/designs/egm_chat.md
2. Work only from user-supplied text and files, or a connected document app the user chooses.
3. Sort each issue as a blocker, a fix to make while proceeding, or a note. Hold only for a blocker.
4. Reply in a fixed order: decision, blockers, fixes, next step.
5. Treat facts stated in the document as true. Recompute totals and dates from the document's own figures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Treating stated facts as true stops the review from flagging a wrong premise. Suggest: do not invent counter-facts, but flag claims the document contradicts or does not support.

Comment thread docs/designs/egm_chat.md
- On strong models, the skill mainly fixes over-blocking. Without it, models hold most sound plans for reasons the plan does not support.
- The skill does not find more real problems. Strong models already find 97 to 100% unaided.
- No single-reply instruction raised recall on a small model (gemini-3.5-flash-lite). Two independent fresh reviews raised it only from 77% to 82%, because both tend to miss the same flaws. Extra catches need separate fresh-context reviewers, which is full EGM.
- On gemini-3.5-flash-lite, the skill lets some real problems through: right decisions on flawed plans were 77 to 95%, against 100% with no skill. The skill should recommend a strong model and full EGM Review for decisions users depend on.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recommendation to suggest a strong model and full EGM Review is not in the Behavior Rules or Required Tests, and the skill cannot see which model runs it in ChatGPT. Is this a requirement, or a known limit to list under risks?

Comment thread docs/designs/egm_chat.md
- **Then:** EGM routes to `egm-review` and retains its isolation checks. A document-only request can still use `egm-in-chat` in that chat.
- **Then:** Repository operations additionally require access to the requested repository and command execution. The skill must not infer either from the desktop label.

## Evaluation Evidence

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About 60 lines here are prompt-tuning data. Could the method and tables move to openai/skills/egm-in-chat/TEST.md or docs/verification/, with a short summary and the pilot gate kept here? Please also link the cases and harness, since rules 3 to 7 rest on them. One version scheme would help too: plugin.json is 0.1.1 and skills have no version field, but the doc mentions 0.1.0, 0.2.0, and 0.3.0.

Comment thread docs/designs/egm_chat.md
- On a capable desktop chat, run `egm-brainstorm`, `egm-draft`, and `egm-review` end to end and verify native fresh-context reviewer evidence. Record `not run` for any missing check. Installation or unit tests alone do not satisfy this gate.
- Connected-document use is tested only when the host exposes it. Uploads remain the fallback.

## Open Questions

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no risks section. A short list would help: full skills visible but unusable on web, weaker models under-blocking (L139), writes to connected files (L43), prompt injection through uploaded documents, and privacy of uploads to a hosted chat.

Comment thread docs/designs/egm_chat.md
- **Date:** 2026-11-16
- **Scope issue:** #414
- **Related:** #460, #419, #438 (`egm-lite` for repo-backed engineering), #371 (24 KB skill load limit)
- **Earlier names:** `egm-lite`, `egm-chat`. This design proposes `/egm-in-chat` (`EGM in Chat` in the OpenAI UI).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: egm-lite is listed as an earlier name of this skill, but it is now a different shipped skill (#438), and L16 adds "Engineering Lite". One sentence on which skill owns the name would help.

@seanstappas

Copy link
Copy Markdown
Collaborator

The skill is well bounded, and keeping its source out of skills/ makes sense. Before I approve, three things need changes: how far the flag removal reaches (L164), the wording and trigger of the capability check (L166), and one merge gate per PR (L216). Separately, #414 needs design-ready before this can merge, and I'll raise that with the other maintainers.

I also ran /egm-review (feature spec stages) on the doc. The inline comments include its main findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation triage: passed Automated Stage-1 PR triage found zero blocking or major issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants