Repository navigation
Conversation
egm-triage-bot
left a comment
There was a problem hiding this comment.
Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.
f02f8b1 to
eebeb8a
Compare
egm-triage-bot
left a comment
There was a problem hiding this comment.
Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.
eebeb8a to
e97e967
Compare
egm-triage-bot
left a comment
There was a problem hiding this comment.
Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.
egm-triage-bot
left a comment
There was a problem hiding this comment.
Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.
Refresh the design branch with the merged EGM Lite implementation and document nine OpenAI skills and eight canonical skills in v0.2.0.
egm-triage-bot
left a comment
There was a problem hiding this comment.
Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.
egm-triage-bot
left a comment
There was a problem hiding this comment.
Automated first-pass review found zero blocking or major issues. Routing to @google/egm-maintainers for maintainer review.
|
|
||
| Store the document skill in `openai/skills/egm-in-chat/`. The OpenAI packer places its runtime files at `skills/egm-in-chat/` alongside the eight canonical skills, including `egm-lite`, using the same path-safety and reproducibility checks. The Claude build and repository-based coding-harness skill discovery retain those eight canonical skills. | ||
|
|
||
| Remove `requires_local_executor` from both OpenAI manifest locations. Update the listing description to distinguish document work from independent review. This prepares a candidate; it does not publish an update to the current desktop-only listing. |
There was a problem hiding this comment.
Removing requires_local_executor reaches further than "prepares a candidate". The Codex catalog installs from the repo root (.agents/plugins/marketplace.json has "source": "./"), and workspace import uses the same path, so those installs lose the flag as soon as #464 merges. docs/codex.md says the flag is there because bootstrap needs shell, and tests/unit/test_harnesses.py asserts it is set.
Can you state this as the main decision, list the affected install paths, and add a desktop egm-bootstrap shell check to the gate at L218? If desktop loses execution, stripping the flag at pack time for the OpenAI target could be the fallback.
|
|
||
| Remove `requires_local_executor` from both OpenAI manifest locations. Update the listing description to distinguish document work from independent review. This prepares a candidate; it does not publish an update to the current desktop-only listing. | ||
|
|
||
| All full skills already read `docs/codex.md` before starting on OpenAI hosts. That document adds a capability check before any workflow intake: readable bundled resources and writable artifacts for full EGM; verified fresh-context native subagents for any Goldfish stage; repository and command access for repository operations. If a requirement is absent, stop that workflow, name the missing capability, and offer `egm-in-chat` when it is installed. Do not silently replace an independent review with same-context critique. |
There was a problem hiding this comment.
This reads as if the check already exists, but #464 adds it. Every full skill loads docs/codex.md only "When running in Codex or ChatGPT Work", so ChatGPT Chat (where the pilot ran, L144) may never read it. Please phrase this as a proposed change and widen that trigger to all OpenAI hosts. The stop rule against same-context Goldfish review is already inline in each full skill, which is worth saying here.
This change also reaches Codex CLI and IDE users, so Required Tests should include a Codex CLI run of brainstorm, draft, and review.
|
|
||
| - **Given:** The same OpenAI package is installed in a chat with readable bundled files, writable artifacts, and verified fresh-context subagents. | ||
| - **When:** The user requests independent review. | ||
| - **Then:** EGM routes to `egm-review` and retains its isolation checks. A document-only request can still use `egm-in-chat` in that chat. |
There was a problem hiding this comment.
All full skills set disable-model-invocation: true, so "EGM routes to egm-review" can only mean telling the user which skill to select. Can you say that here so the design does not imply an automatic handoff?
| - Unit tests check that the existing OpenAI archive has the same `egm` identity, contains all nine skills and their runtime assets, omits `requires_local_executor` in both manifests, and is reproducible. The Claude archive has eight skills. Both archives must include `egm-lite`; only OpenAI includes `egm-in-chat`. Missing or symlinked document-skill files fail before output. | ||
| - `ruff check .` and `python -m pytest tests/unit` pass. | ||
| - Offline package validation passes for the combined OpenAI package. | ||
| - Before merge or publication, install the exact same archive in clean ChatGPT web and desktop chats. Record archive checksum, package version, host version or date, installation result, skill discovery, and observed capabilities separately for each surface. |
| The skill tied the plain model on five of six pairs and lost one. | ||
| No reply invented facts or claimed independent review. | ||
| ChatGPT Work listed all packaged skills; the UI did not show whether the skill file was read. | ||
| The pilot must be repeated on version 0.3.0 before Stage 2 merges. |
There was a problem hiding this comment.
This gate needs a pass bar. The only on-host result so far is 5 ties and 1 loss, so as written it cannot fail. Something like "no losses against the plain model and no independence claims across N blind pairs" would work.
| 2. Work only from user-supplied text and files, or a connected document app the user chooses. | ||
| 3. Sort each issue as a blocker, a fix to make while proceeding, or a note. Hold only for a blocker. | ||
| 4. Reply in a fixed order: decision, blockers, fixes, next step. | ||
| 5. Treat facts stated in the document as true. Recompute totals and dates from the document's own figures. |
There was a problem hiding this comment.
Treating stated facts as true stops the review from flagging a wrong premise. Suggest: do not invent counter-facts, but flag claims the document contradicts or does not support.
| - On strong models, the skill mainly fixes over-blocking. Without it, models hold most sound plans for reasons the plan does not support. | ||
| - The skill does not find more real problems. Strong models already find 97 to 100% unaided. | ||
| - No single-reply instruction raised recall on a small model (gemini-3.5-flash-lite). Two independent fresh reviews raised it only from 77% to 82%, because both tend to miss the same flaws. Extra catches need separate fresh-context reviewers, which is full EGM. | ||
| - On gemini-3.5-flash-lite, the skill lets some real problems through: right decisions on flawed plans were 77 to 95%, against 100% with no skill. The skill should recommend a strong model and full EGM Review for decisions users depend on. |
There was a problem hiding this comment.
The recommendation to suggest a strong model and full EGM Review is not in the Behavior Rules or Required Tests, and the skill cannot see which model runs it in ChatGPT. Is this a requirement, or a known limit to list under risks?
| - **Then:** EGM routes to `egm-review` and retains its isolation checks. A document-only request can still use `egm-in-chat` in that chat. | ||
| - **Then:** Repository operations additionally require access to the requested repository and command execution. The skill must not infer either from the desktop label. | ||
|
|
||
| ## Evaluation Evidence |
There was a problem hiding this comment.
About 60 lines here are prompt-tuning data. Could the method and tables move to openai/skills/egm-in-chat/TEST.md or docs/verification/, with a short summary and the pilot gate kept here? Please also link the cases and harness, since rules 3 to 7 rest on them. One version scheme would help too: plugin.json is 0.1.1 and skills have no version field, but the doc mentions 0.1.0, 0.2.0, and 0.3.0.
| - On a capable desktop chat, run `egm-brainstorm`, `egm-draft`, and `egm-review` end to end and verify native fresh-context reviewer evidence. Record `not run` for any missing check. Installation or unit tests alone do not satisfy this gate. | ||
| - Connected-document use is tested only when the host exposes it. Uploads remain the fallback. | ||
|
|
||
| ## Open Questions |
There was a problem hiding this comment.
There is no risks section. A short list would help: full skills visible but unusable on web, weaker models under-blocking (L139), writes to connected files (L43), prompt injection through uploaded documents, and privacy of uploads to a hosted chat.
| - **Date:** 2026-11-16 | ||
| - **Scope issue:** #414 | ||
| - **Related:** #460, #419, #438 (`egm-lite` for repo-backed engineering), #371 (24 KB skill load limit) | ||
| - **Earlier names:** `egm-lite`, `egm-chat`. This design proposes `/egm-in-chat` (`EGM in Chat` in the OpenAI UI). |
There was a problem hiding this comment.
Nit: egm-lite is listed as an earlier name of this skill, but it is now a different shipped skill (#438), and L16 adds "Engineering Lite". One sentence on which skill owns the name would help.
|
The skill is well bounded, and keeping its source out of I also ran |
Summary of Key Changes
Refs #414. Part 1 of 2 in stack #472: #435 design → #464 implementation.
Design ChatGPT web support in the existing EGM plugin and listing, using one OpenAI archive for web and desktop.
egm-in-chatfor document brainstorming, drafting, and explicitly same-context review.openai/skills/, preserving shared coding-harness discovery. OpenAI has nine skills; Claude and repository installs have eight, including engineering Lite from feat(skills): implement EGM Lite rapid engineering protocol (Stage 2) #438.Goals and Non-Goals are in
docs/designs/egm_chat.md. The shared/egmrouter has been removed from this proposal and extracted to stack #476: #474 design → #475 implementation, for #327. It is not a dependency of this stack.Review this design before #464. The implementation prepares v0.2.0 and the checklist for updating the existing OpenAI listing. Merging the stack does not publish the plugin. Individual PR auto-merge is disabled for native stack membership.
Contribution Type
Track: Feature / Architectural Change -- Stage 1: Design Doc PR
Verification
egm-reviewandegm-resolve: not run.Human Accountability Affirmation