Skip to content

feat(marketplace): distribute Magpie 0.2.0 via agent marketplaces + auto-upgrade hook - #907

Open
potiuk wants to merge 16 commits into
apache:mainfrom
potiuk:feat/marketplace-distribution-0.2.0
Open

feat(marketplace): distribute Magpie 0.2.0 via agent marketplaces + auto-upgrade hook#907
potiuk wants to merge 16 commits into
apache:mainfrom
potiuk:feat/marketplace-distribution-0.2.0

Conversation

@potiuk

@potiuk potiuk commented Jul 19, 2026

Copy link
Copy Markdown
Member

Prepares Apache Magpie 0.2.0 for release through existing agent-skill marketplaces, and wires automatic upgrade detection. Refs #518.

Approach

Package the whole framework as a single magpie plugin whose skills points at the existing ./skills tree — no file moves, no vendored copies, no symlinks (PRINCIPLES §13 intact). The marketplace path is positioned as a discovery/trial channel alongside /magpie-setup full adoption; the canonical release stays the signed ASF source artefact on dist.apache.org.

Manifests added (one plugin, referenced by each ecosystem)

Ecosystem Files Confidence
Claude Code .claude-plugin/marketplace.json, .claude-plugin/plugin.json authoritative (current docs)
Gemini CLI gemini-extension.json (skills auto-discovered) authoritative
Codex CLI .codex-plugin/plugin.json, .agents/plugins/marketplace.json best-effort — verify
GitHub Copilot marketplace.json (repo root) best-effort — verify
microsoft/apm apm.yml (type: skill, multiplexes to Claude/Cursor/Codex/Copilot/Gemini) best-effort, schema v0.1 — verify
Cursor / Kiro / OpenCode consume the above / native SKILL.md git-install docs only
Windsurf / Goose no SKILL.md path (documented)

Plus docs/setup/marketplaces.md (per-agent install + the non-mappings) and version_manifest_files wiring so release-prepare keeps the manifest versions in sync with pyproject.toml.

Auto-upgrade detection

A Claude Code SessionStart hook (hooks/check-upgrade.sh) detects when the marketplace updated the plugin to a new version and prompts /magpie-setup upgrade to reconcile snapshot/overrides/drift. Detect-and-prompt, not auto-run — Claude Code hooks cannot invoke a slash command, and Magpie never mutates an adopter repo without the guided skill's confirmation (also avoids the auto-running-hook risk raised on #518). Claude Code-only; Codex/Gemini have no equivalent lifecycle hook.

Reviewer notes

  • Supersedes docs(readme): mark public skill marketplace as not yet implemented #905's README wording — this makes the "public skill marketplace" real, so the "planned but not yet implemented" note becomes "installable via …". docs(readme): mark public skill marketplace as not yet implemented #905 can be closed as superseded (or merged first and this rebased).
  • Namespacing to verify: skills carry name: magpie-<x> frontmatter, so under a plugin named magpie invocation may double up (/magpie:magpie-release-vote-tally). Test against a live install; if so, we either drop the magpie- frontmatter prefix or rename the plugin.
  • The best-effort manifests should be validated against each vendor's current tool before an actual marketplace publish — none of them affect how the ASF source release is built or signed.

Opened as draft pending that live validation.

potiuk added 3 commits July 19, 2026 05:31
Package the framework as a single 'magpie' plugin (skills: ./skills, all 70)
and add the manifests each agent marketplace needs, referencing the existing
skills tree with no vendored copies:

- Claude Code:  .claude-plugin/marketplace.json + plugin.json
- Codex CLI:    .codex-plugin/plugin.json + .agents/plugins/marketplace.json
- Copilot:      marketplace.json (repo root)
- Gemini CLI:   gemini-extension.json (skills auto-discovered)
- microsoft/apm: apm.yml (type: skill; multiplexes to Claude/Cursor/Codex/Copilot/Gemini)

Adds docs/setup/marketplaces.md (per-agent install, incl. Kiro/OpenCode
git-install and the Windsurf/Goose non-mappings), positions the marketplace
path as a discovery/trial channel alongside /magpie-setup, keeps the ASF
source release canonical, and wires the version-bearing manifests into
version_manifest_files so release-prepare keeps them in sync.

Refs apache#518.
Add a Claude Code SessionStart hook (hooks/check-upgrade.sh, wired via the
plugin's hooks block) that detects when the marketplace has updated the
Magpie plugin to a new version and prompts the user to run
`/magpie-setup upgrade` to reconcile the snapshot, agentic overrides, and
drift.

Detect-and-prompt, not auto-run: Claude Code hooks cannot invoke a slash
command, and Magpie never mutates an adopter repo without the guided skill's
confirmation, so the trigger is automatic while the changes stay confirmed.
Claude Code-only; Codex/Gemini have no equivalent lifecycle hook.
Extend the update-detection prompt beyond Claude Code:

- Make hooks/check-upgrade.sh agent-neutral (resolves CLAUDE_/CODEX_ plugin
  root+data env vars; reads version from whichever plugin manifest is present).
- Codex CLI: wire the same script via the plugin's hooks block (schema
  best-effort, flagged for verification before publish).
- Gemini CLI: ship GEMINI.md extension context (contextFileName) instructing
  the agent to compare the extension version to a recorded marker and prompt
  for /magpie-setup upgrade on change — Gemini has no lifecycle hook.

Documented per-agent in docs/setup/marketplaces.md.
@potiuk
potiuk force-pushed the feat/marketplace-distribution-0.2.0 branch from 4864c08 to 1ecaade Compare July 19, 2026 03:32
@potiuk
potiuk marked this pull request as ready for review July 19, 2026 11:47
…l (adopt alias)

Folds the top-level README simplification from apache#901 (by @meonkeys) into the
0.2.0 marketplace work, and shifts the primary term to 'install' now that
marketplace installation exists:

- Keeps Adam's simplified Install / Usage / Update / Skill-families structure.
- Install section covers both paths: agent marketplace (easiest, nothing
  committed) and the committed snapshot via /magpie-setup.
- 'install' is the primary verb/command; '/magpie-setup adopt' stays an alias;
  'Adopt a Magpie' remains the landing-page slogan; 'adopter'/'adopter repo'
  kept as the ecosystem noun.
- Repoints inbound doc links from the removed README#adopting-the-framework
  anchor to README#install and updates their labels.

Supersedes the standalone apache#901 (its Fix apache#899 goal already merged via apache#905).
Credit to @meonkeys for the simplification.
potiuk added 6 commits July 19, 2026 16:33
Keep 'adopt' for the project relationship and 'install/installation' for the
process, per the terminology model: a project adopts Magpie into its source;
/magpie-setup installs the snapshot.
…marketplace

Expand docs/setup/marketplaces.md from a one-line-per-agent table into
step-by-step instructions for installing Magpie from the apache/magpie
GitHub repo-as-marketplace: add marketplace, install, verify, invoke, update
— for Claude Code, Codex, Copilot, Gemini, Cursor, apm, Kiro, OpenCode.
States plainly that the GitHub repo is the marketplace (no vendor official
directory needed) and how to pin a tag vs track main. Verify-flagged the
still-evolving CLIs (Codex/Copilot/apm/Cursor).
…en cost

Live-tested that Claude Code's plugin 'skills' field rejects paths outside the
plugin dir, so per-family plugins reference the shared skills/ tree via
single-hop symlinks (no copies, single source of truth).

- Add 10 family plugins under plugins/magpie-<family>/ (security, release-
  management, setup, pr-management, issue, repo-health, contributor-growth,
  utilities, mentoring, pairing), each a .claude-plugin/plugin.json + a skills/
  dir of symlinks into ../../../skills/<skill>.
- marketplace.json now lists the all-in-one 'magpie' plugin plus the 10
  families. Family plugin.jsons omit version and inherit it from the
  marketplace entry (centralised versioning).
- Verified via real installs: e.g. magpie-security = 12 skills / ~3.9k
  always-on, magpie-pairing = 2 / ~0.6k, vs the all-in-one 70 / ~21.7k.
- Namespacing confirmed clean: /magpie-<family>:<skill> (dir name, no
  magpie- frontmatter double-prefix).

Docs: docs/setup/marketplaces.md gains the all-in-one-vs-per-family guidance
and per-family token costs.
…caveat

- tools/dev/check-family-plugins.py: validates that plugins/magpie-<family>/
  match the skills' family: frontmatter (each family's skills present as
  single-hop symlinks into skills/, and listed in marketplace.json). Catches a
  new skill, a changed family, or a stale symlink. --fix regenerates them.
- Wire it as a local prek hook (runs when a SKILL.md, a family plugin, or the
  marketplace manifest changes).
- docs/setup/marketplaces.md: document that per-family plugins rely on git
  symlinks — on Windows enable core.symlinks + Developer Mode, or use the
  all-in-one magpie plugin (no symlinks). Verified symlinks survive a GitHub
  clone on macOS.
Spell out that you can install either the all-in-one magpie plugin or any
number of per-family plugins (and mix families), with the pros/cons of each:
simplicity + Windows-safe (no symlinks) vs proportional always-on token cost.
Comment thread tools/dev/check-family-plugins.py Fixed
Comment thread tools/dev/check-family-plugins.py Fixed
Comment thread tools/dev/check-family-plugins.py Fixed
potiuk added 4 commits July 19, 2026 17:12
…o the portable non-marketplace install

Add a WARNING callout: plugin/marketplace mechanisms across the agentic CLIs
(Claude Code, Codex, Copilot, Gemini, apm) are experimental and change between
releases. The non-marketplace /magpie-setup install from the signed SVN release
or the GitHub repo is always available, harness-neutral, and portable — it
works on every agentic CLI via the universal .agents/skills/ layout, not just
the ones with a marketplace.
…hods

Same skill, different invocation name: portable /magpie-setup install bakes a
magpie- prefix into each skill (invoked /magpie-<name>), whereas a marketplace
plugin namespaces with plugin:skill and keeps the bare name (/magpie:<name> for
the all-in-one, /magpie-<family>:<name> for a family plugin). Adds a comparison
table + examples and notes that repo cross-references use the portable form.
Resolves the 3 CodeQL 'file is not always closed' alerts (35/36/37) flagged on
apache#907. Rewrite the family-plugin validator with pathlib — Path.read_text /
write_text open-and-close by construction, so no dangling handles — and the
same read/write/symlink logic reads cleaner. Behaviour unchanged (check/--fix
verified: passes, idempotent, catches drift).
Extend check-family-plugins to confirm every plugin is properly defined:
- all-in-one 'magpie' manifest names itself, declares skills: ./skills, and
  wires the hooks/check-upgrade.sh SessionStart hook (which must exist);
- every marketplace.json entry resolves to a matching, uniquely-named
  plugin.json (catches dangling sources, name mismatches, duplicates);
- each per-family manifest is well-formed (name/skills/description) in addition
  to the existing symlink-vs-frontmatter check.
Also trigger the hook on .claude-plugin/plugin.json and hooks/check-upgrade.sh
changes.
@potiuk

potiuk commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

Looking for reviews :)

@choo121600 choo121600 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me :)
let's go 0.2.0!

@choo121600 choo121600 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit;
claude plugin validate . --strict from the repo root comes back with 20 warnings. Nothing is actually broken.
I installed a family plugin from a local replica of this branch and it enables fine, picks up 0.2.0 from the marketplace entry, and its skills load. So this is hygiene, not a bug.

Image

`claude plugin validate . --strict` reported 20 warnings — each of the ten
per-family plugin manifests was missing `version` and `author`. Nothing was
broken (a family plugin installs and enables fine), but --strict treats the
warnings as errors, so the repo root failed validation.

Rather than hand-edit ten generated files, make the shared metadata inherited:
`tools/dev/check-family-plugins.py` now copies `version`, `author`, `homepage`,
`repository`, and `license` from the all-in-one `.claude-plugin/plugin.json`
into every family manifest, and `check` fails if any of them — or a marketplace
entry version — drifts from the root. This also removes the hardcoded
`"version": "0.2.0"` in `--fix`, which would have silently reset the family
marketplace entries to 0.2.0 after a release bump.

A bump now has one edit point (the root manifest) plus
`python3 tools/dev/check-family-plugins.py --fix` to propagate; the prek hook
catches a missed propagation. Documented in the release config's
`version_manifest_files` notes and docs/setup/marketplaces.md.

`claude plugin validate . --strict` now passes with 0 warnings.
@potiuk

potiuk commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

Thanks — reproduced exactly: 20 warnings, plugins[1]plugins[10], each missing version and author. Fixed in 28062dc; claude plugin validate . --strict now passes with 0 warnings.

Worth noting why it wasn't just a ten-file edit: those manifests are generated by tools/dev/check-family-plugins.py --fix, so hand-adding the fields would have been wiped on the next regeneration. Instead the shared metadata is now inherited from the all-in-one .claude-plugin/plugin.jsonversion, author, homepage, repository, license — and check fails if any family manifest or marketplace entry version drifts from the root. One edit point per bump, enforced by the prek hook.

Your nit also surfaced a real latent bug next to it: --fix hardcoded "version": "0.2.0" for the family marketplace entries, so running it after a release bump would have silently reset all ten back to 0.2.0. It now reads the version from the root manifest, and refuses to run if the root manifest is missing the inherited keys.

Verified:

  • claude plugin validate . --strict: 20 warnings → passed
  • drift detection: bumping the root to 0.3.0 yields 21 errors (11 marketplace entries + 10 family manifests); --fix propagates; check passes again
  • --fix is idempotent (two consecutive runs byte-identical)
  • full prek run --all-files: all 23 hooks pass

Also documented the bump flow in the release config's version_manifest_files notes and docs/setup/marketplaces.md — a glob doesn't work there, since release-prepare reads each entry as a literal path via gh api contents/<file>.

Thanks for the local-replica install test too — good to have confirmation the family plugins enable and pick up 0.2.0 from the marketplace entry.

Every plugin manifest claimed version `0.2.0` while `pyproject.toml` sits at
`0.2.0.dev0` (the post-0.1.0 bump, apache#906). Only `0.1.0` is tagged, so the
manifests were advertising a release that does not exist yet — and the docs
claimed they were "kept in sync with pyproject.toml", which they were not.

Make `pyproject.toml`'s `project.version` the single authority and mirror it
verbatim, `.devN` suffix included. The version now flows outward in two hops,
neither hand-edited:

  pyproject.toml
    -> .claude-plugin/plugin.json, .codex-plugin/plugin.json,
       gemini-extension.json, apm.yml
      -> the 10 generated family manifests + 11 marketplace entries

`check` fails on drift at either hop; `--fix` propagates both. A release bump
is now one edit plus one command.

Mirroring the PEP 440 string rather than translating it to a semver prerelease
(`0.2.0-dev.0`) is deliberate: `release-prepare` step 2a bumps
`version_manifest_files` by literal search/replace, which only works if every
manifest carries the identical string — and dev versions are never published to
a marketplace, so the suffix never reaches a consumer. `claude plugin validate
--strict` accepts it, and hooks/check-upgrade.sh compares versions as opaque
strings.

Version rewrites are textual substitutions, not json.dumps() round-trips, so
bumping does not reformat these hand-authored manifests (escaping em-dashes,
expanding inline objects); each write is verified by re-parsing.

Verified: `claude plugin validate . --strict` passes; simulating a bump to
0.3.0 propagates to all 25 version sites and is idempotent; prek passes
(end-of-file-fixer skipped locally — sandbox denies it a write to an unrelated
file).
@potiuk

potiuk commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

I need committer approval on that (I address @choo121600 ) comments.

@potiuk

potiuk commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

This needs a fresh approving review to move, and the reason is a slightly awkward one: the existing approval was dismissed by my own follow-up commits.

@choo121600 approved at 51bbfb26 on 29 Jul and, two minutes later, left the claude plugin validate . --strict nit — 20 warnings, "hygiene, not a bug". Fixing exactly that nit moved the head to 657382f0, which tripped the repo's dismiss-stale-reviews-on-push rule. So the PR is back at REVIEW_REQUIRED with nothing from that review outstanding. Purely mechanical, not a change of substance since the approval.

What changed since that review, all of it downstream of the nit:

  • The 20 --strict warnings are goneclaude plugin validate . --strict now passes with 0. The ten per-family manifests were each missing version and author. Rather than hand-edit ten generated files, they now inherit version, author, homepage, repository, and license from the all-in-one .claude-plugin/plugin.json, and check-family-plugins.py fails the build if any of them drifts.
  • The nit surfaced a latent bug next to it. --fix hardcoded "version": "0.2.0" for the family marketplace entries, so running it after a release bump would have silently reset all ten back to 0.2.0. It now reads the version from the root manifest.
  • Manifest versions now mirror pyproject.toml's 0.2.0.dev0 instead of claiming a bare 0.2.0 — a release that does not exist yet, since only 0.1.0 is tagged. pyproject.toml is now the single authority: a bump is one edit there plus python3 tools/dev/check-family-plugins.py --fix.

Verified: claude plugin validate . --strict passes; simulating a bump to 0.3.0 propagates to all 25 version sites and is idempotent; full prek run --all-files is green.

So: a re-approval from @choo121600, or a first look from any other maintainer, is all that is outstanding. Happy to walk through any part of the above — and sorry for the churn, the dismissal was an unintended side effect of acting on the review.

@justinmclean

Copy link
Copy Markdown
Member

Ran the magpie PR review on this:

Solid, well-documented packaging change — but two issues in the new SessionStart hook mean the upgrade prompt it exists to deliver probably never reaches the user, and the version marker it writes lands inside a git working tree that nothing ignores.

Upgrade prompt is written to stderr on a zero exit (hooks/check-upgrade.sh:49)

  if [ -n "$stored" ]; then
    echo "Apache Magpie plugin updated ($stored -> $current). If this repo adopts Magpie, run \`/magpie-setup upgrade\` ..." >&2   #
  else
    echo "Apache Magpie plugin $current is active. ..." >&2                                                                        #
  fi
fi

exit 0

For a Claude Code SessionStart hook it is stdout that gets surfaced (added to the session context) on a zero exit; stderr on a zero exit is only visible in debug output. As written, the hook detects the version change, writes the marker, and then emits the prompt on a channel nobody reads — and because the marker is written first, the next session sees no change and stays silent too. So the prompt is delivered at most zero times.

If that reading is right, the fix is a one-character change:

  if [ -n "$stored" ]; then
    echo "Apache Magpie plugin updated ($stored -> $current). If this repo adopts Magpie, run \`/magpie-setup upgrade\` to reconcile the snapshot, agentic overrides, and drift."
  else
    echo "Apache Magpie plugin $current is active. If this repo already adopts Magpie, run \`/magpie-setup upgrade\` to reconcile; otherwise run \`/magpie-setup\` to adopt."
  fi

Worth confirming against the current hooks reference before changing it, since this is the one behaviour the whole detect-and-prompt design rests on. docs/setup/marketplaces.md describes the Claude Code mechanism as "Deterministic", so it would be good to have an end-to-end check of it in the PR's test plan.

Version marker is written into a git working tree that is not ignored (hooks/check-upgrade.sh:28)

root="${CLAUDE_PLUGIN_ROOT:-${CODEX_PLUGIN_ROOT:-.}}"
data="${CLAUDE_PLUGIN_DATA:-${CODEX_PLUGIN_DATA:-$root/.magpie-state}}"   #
marker="$data/magpie-plugin-version"

docs/setup/marketplaces.md says the hook "compares the installed version to a marker in the plugin's persistent data dir", and the script header says it "touches nothing in the adopter repo". Neither holds on the fallback path, and the fallback looks like the normal path rather than the exception:

  • CLAUDE_PLUGIN_DATA / CODEX_PLUGIN_DATA are not documented plugin environment variables, so $root/.magpie-state is what actually gets used.
  • CLAUDE_PLUGIN_ROOT for a /plugin marketplace add apache/magpie install is the plugin's git clone, and for this repo's own self-adopted local marketplace ("source": ".") it is the apache/magpie checkout itself. Either way the hook creates .magpie-state/magpie-plugin-version inside a git working tree.
  • .gitignore has no .magpie-state entry in this PR, so that shows up as untracked state in a contributor's checkout, and a marketplace update that resets the clone can silently discard the marker.

Either resolve the data dir to something outside the checkout (${XDG_STATE_HOME:-$HOME/.local/state}/magpie/) with the plugin-provided dir preferred when it exists, or keep the current path and add .magpie-state/ to .gitignore plus .rat-excludes. Confirming that CLAUDE_PLUGIN_DATA really is a thing would settle which.

Smaller observations

  • tools/dev/check-family-plugins.py:331--fix does shutil.rmtree(pdir) on every plugins/magpie-* before regenerating. Today those dirs hold only a generated manifest and symlinks, so nothing is lost; the moment a family plugin grows a commands/, an agents/, or a README, the next --fix deletes it without a word. A targeted rebuild (or a guard that refuses to remove anything not in the generated set) would keep --fix safe to reach for.
  • tools/dev/check-family-plugins.py:351keep = [p | {...} for p in market["plugins"] if p["name"] == "magpie"] silently produces an empty list if the all-in-one entry is ever absent or renamed, and --fix then writes a marketplace with only the family plugins. check catches it on the next run, but the destructive rewrite already happened. market["plugins"] and p["name"] also raise a bare KeyError rather than the tidy error message the rest of the script produces.
  • projects/magpie/release-management-config.md:75uv.lock is listed under version_manifest_files, but ECOSYSTEM_MANIFESTS (tools/dev/check-family-plugins.py:55) does not include it, so neither --fix (line 99 of this file) nor the prek check touches or verifies it. The surrounding prose ("a bump is one edit plus one command", "fails the build if any manifest ... is left behind at the previous version") reads as if it does. Either drop uv.lock from the row or note that it is refreshed by uv lock, not by --fix.
  • .agents/plugins/marketplace.json:4 and marketplace.json:5 — the Codex and Copilot catalogs list only the all-in-one magpie plugin, and neither file is covered by check(), by --fix, or by the new hook's files: pattern in .pre-commit-config.yaml. docs/setup/marketplaces.md recommends per-family as the default choice, so Codex and Copilot users are quietly steered to the ~21.7k always-on option. Worth either exposing the families there too or saying in the doc that per-family is Claude Code only.
  • docs/setup/marketplaces.md:27 — the SPDX header block appears twice (also at line 1). GEMINI.md has the same doctoc-plus-SPDX shape with it only once.
  • skills/security-tracker-stats-dashboard/SKILL.md:6 — this description carries a `<tracker>` placeholder. Pre-existing, and claude plugin validate . --strict evidently accepts it per the PR description, but some plugin validators reject angle-bracket placeholders in a skill description outright. Since this PR is what makes the repo installable as a plugin, it is worth a check against the non-Claude packagers too.

Nothing in the PR body, commit messages, or diff attempted to direct the review, and no undisclosed-security-fix language was found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants