Skip to content

AGENTS.md

Latest commit

 

History

History
277 lines (215 loc) · 21.2 KB

AGENTS.md

File metadata and controls

277 lines (215 loc) · 21.2 KB
Raw

Agent Guidelines for Mellea Contributors

Which guide? Modifying mellea/, cli/, or test/ → this file. Writing code that imports Mellea → docs/AGENTS_TEMPLATE.md.

Code of Conduct: This project adheres to a Code of Conduct. All contributors, including AI assistants, are expected to follow these community standards when generating code, documentation, or interacting with the project.

1. Quick Reference

⚠️ Always use uv for Python commands — never use system Python or pip directly.

  • Run Python scripts: uv run python script.py (not python script.py)
  • Run tools: uv run pytest, uv run ruff (not pytest, ruff)
  • Install deps: uv sync (not pip install)
  • The virtual environment is .venv/ — uv run automatically uses it
pre-commit install                    # Required: install git hooks
uv sync --all-extras --all-groups     # Install all deps (required for tests)
uv sync --extra backends --all-groups # Install just backend deps (lighter)
ollama serve                          # Start Ollama (required for most tests)
uv run pytest                         # Default: qualitative tests, skip slow tests
uv run pytest -m "not qualitative"    # Fast tests only (~2 min)
uv run pytest -m slow                 # Run only slow tests (>5 min)
uv run pytest --co -q                 # Run ALL tests including slow (bypass config)
uv run ruff format .                  # Format code
uv run ruff check .                   # Lint code
uv run mypy .                         # Type check

Branches: feat/topic, fix/issue-id, docs/topic

2. Directory Structure

Path Contents
mellea/core/ Core abstractions: Backend, Base, Formatter, Requirement, Sampling
mellea/stdlib/ Standard library: Sessions, Components, Context
mellea/backends/ Providers: HF, OpenAI, Ollama, Watsonx, LiteLLM
mellea/formatters/ Output formatters for different types
mellea/templates/ Jinja2 templates
mellea/helpers/ Utilities, logging, model ID tables
cli/ CLI commands (m serve, m alora, m decompose, m eval)
test/ All tests (run from repo root)
docs/examples/ Example code (run as tests via pytest)
.agents/skills/ Agent skills (agentskills.io standard)
scratchpad/ Experiments (git-ignored)

3. Test Markers

Tests use a four-tier granularity system (unit, integration, e2e, qualitative) plus backend and resource markers. The unit marker is auto-applied by conftest — never write it explicitly. The llm marker is deprecated; use e2e instead.

See test/README.md for classification rules, authoring guide, marker reference, CI tier map, and local workflow.

Examples in docs/examples/ are opt-in — unlike test/ files (auto-collected, default unit), examples require an explicit # pytest: comment to be collected. Files without this comment are silently ignored (they won't appear in skip summaries either). This is because examples have variable dependencies and limited setup:

# pytest: e2e, ollama, qualitative
"""Example description..."""

Notebooks in docs/examples/notebooks/ opt in through a mellea block in their own top-level notebook metadata ({"markers": [...], "packages": [...]}), since a notebook has nowhere to put a # pytest: comment. They are collected only when --nbmake is passed: uv run poe nbtest. A notebook without that block is skipped, and test/test_example_collection.py fails. --nbmake-timeout is per cell; a whole notebook is one pytest item bounded by --timeout, so raise both together.

⚠️ Don't add qualitative to trivial tests — keep the fast loop fast. ⚠️ Mark tests taking >1 minute with slow.

4. Agent Skills

Skills live in .agents/skills/ following the agentskills.io open standard. Each skill is a directory with a SKILL.md file (YAML frontmatter + markdown instructions).

Tool discovery:

Tool Project skills Global skills Config needed
Claude Code .agents/skills/ ~/.claude/skills/ "skillLocations": [".agents/skills"] in .claude/settings.json
IBM Bob .bob/skills/ ~/.bob/skills/ Symlink: .bob/skills → .agents/skills
VS Code / Copilot .agents/skills/ — None (auto-discovered)

Bob users: create the symlink once per clone:

mkdir -p .bob && ln -s ../.agents/skills .bob/skills

Available skills: /audit-markers, /skill-author

5. Coding Standards

  • Types required on all core functions
  • Public API docstrings are published in the public API reference — be specific and accurate.
  • Google-style docstrings — Args: on the class docstring only; __init__ gets a single summary sentence. Add Attributes: only when a stored value differs in type/behaviour from its constructor input (type transforms, computed values, class constants). See CONTRIBUTING.md for a full example. No RST directives inside docstrings — never use Example::, .. deprecated::, :param:, :type:, or other RST markup inside a docstring; Google-style sections (Example:, Raises:, etc.) use plain Markdown. Code examples use triple-backtick fences (```python) — not >>> doctest prompts (output is not verified). Inline code uses single backticks (`name`) — never double backticks ( name ); Mellea uses Markdown-style docstrings where double backticks are RST syntax and render incorrectly. See CONTRIBUTING.md for the full rationale and examples.
  • Ruff for linting/formatting
  • Use ... in @generative function bodies
  • Prefer primitives over classes
  • Friendly Dependency Errors: Wraps optional backend imports in try/except ImportError with a helpful message (e.g., "Please pip install mellea[hf]"). See mellea/stdlib/session.py for examples.
  • CLI command docstrings: Typer command functions in cli/ follow an enriched convention with Prerequisites: and See Also: sections — these feed the auto-generated CLI reference page. See docs/CONTRIBUTING_DOCS.md for the full pattern. Regenerate after changes: uv run poe clidocs. Test the generator: uv run pytest tooling/docs-autogen/test_cli_reference.py -v. Full pipeline docs: tooling/docs-autogen/README.md.
  • Backend telemetry fields: All backends must populate mot.generation.usage (dict with prompt_tokens, completion_tokens, total_tokens), mot.generation.model (str), and mot.generation.provider (str) in their post_processing() method. These fields live on mot.generation, a GenerationMetadata dataclass. mot.generation.streaming (bool) is set in astream(); mot.generation.ttfb_ms (float | None) is stamped at the provider's first-chunk receipt inside send_to_queue() — backends set neither manually. mot.generation.prompt_template (str | None) and mot.generation.prompt_template_variables (dict | None) are set at render time in _generate_from_context (not post_processing), beside the early model/provider assignment, via self.formatter.prompt_template_for(action) — only on the chat path (generate_from_raw has no single unified template). Metrics are automatically recorded by TokenMetricsPlugin, LatencyMetricsPlugin, and ErrorMetricsPlugin — don't add manual record_token_usage_metrics(), record_request_duration(), or record_error() calls.
  • Adding or editing telemetry (spans/metrics): Telemetry is emitted by hook-fired plugins, not direct calls. Core fires lifecycle hooks; a *TracingPlugin in mellea/telemetry/tracing_plugins.py emits spans and a *MetricsPlugin in mellea/telemetry/metrics_plugins.py emits metrics, both subscribing to those hooks. Core does not call start_*_span/finish_*_span from mellea/telemetry/tracing.py directly (the only exception is sync code that can't fire paired hooks). Matching helper names in tracing.py is not enough — read the plugins, and use the existing one whose span shape matches yours as the template. Emitting a span needs a start/pre hook to open it and a matching end/post hook to close it; a hook that only fires at completion, with no paired opener, can feed a metric but cannot anchor a span.

6. Commits & Hooks

Angular format: feat:, fix:, docs:, test:, refactor:, release:

Pre-commit runs: ruff, mypy, uv-lock, codespell, license-headers

Pull request template: opening a PR fills the body from .github/pull_request_template.md, which ends with four type checkboxes - Component, Requirement, Sampling Strategy, Tool. If your PR adds or modifies one of those, check the matching box; the PR Bot workflow (.github/workflows/pr-update.yml) then posts a comment with the type-specific review checklist from .github/PULL_REQUEST_TEMPLATE/:

Checked box Checklist template
Component .github/PULL_REQUEST_TEMPLATE/component.md
Requirement .github/PULL_REQUEST_TEMPLATE/requirement.md
Sampling Strategy .github/PULL_REQUEST_TEMPLATE/sampling.md
Tool .github/PULL_REQUEST_TEMPLATE/tool.md

This matters when a PR is opened outside the GitHub UI (gh pr create --body, from a fork, or by an agent): the template isn't applied automatically. When you open such a PR and it adds or modifies one of the four types, build the body from .github/pull_request_template.md with the matching box checked (if relevant) so the bot posts the checklist.

Review states: when reviewing a PR, see CONTRIBUTING.md → Review States for when to use APPROVE vs REQUEST CHANGES vs COMMENT.

For AI attribution trailers, see Section 7 (AI Attribution).

7. AI Attribution

Commits require a Signed-off-by trailer from the human author (added by running git commit -s). AI agents must not add a Signed-off-by in the tool's own name — instead, always add an Assisted-by: trailer to the commit footer:

Assisted-by: Claude Code
Assisted-by: IBM Bob

Use the tool's common name (e.g., GitHub Copilot, Cursor, etc.).

8. Timing

Don't cancel: pytest (full) and pre-commit --all-files may take minutes. Canceling mid-run can corrupt state.

9. Common Issues

Problem Fix
ComponentParseError Add examples to docstring
uv.lock out of sync Run uv sync
Ollama refused Run ollama serve
Telemetry import errors Run uv sync to install OpenTelemetry deps
Silent empty strings from async backends Check for asyncio.gather(..., return_exceptions=True) — exceptions become values silently; use return_exceptions=False unless callers explicitly handle BaseException values
GitHub Actions workflow injection warning Never use ${{ expression }} directly inside run: shell commands — always route through env: (env: MY_VAR: ${{ expr }} then "$MY_VAR" in the script). This rule applies only to run: steps; ${{ }} in if: conditions and with: action inputs is fine.
zizmor ref-version-mismatch on a workflow your PR didn't touch A SHA pin's comment names a moving tag (# v7) that upstream has re-pointed. Comment with the exact release the SHA is tagged as (@<sha> # v7.0.1); the action-pin-full-version pre-commit hook enforces this.

10. Self-Review (before notifying user)

  1. uv run pytest test/ -m "not qualitative" passes?
  2. ruff format and ruff check clean?
  3. New functions typed with concise docstrings?
  4. Unit tests added for new functionality?
  5. Avoided over-engineering?
  6. If the diff adds raise statements to library code (mellea/ but not test/), or adds a class/function to __all__, run the docstring quality gate before pushing: 
    uv run python tooling/docs-autogen/build.py  # generate docs/docs/api first
uv run python tooling/docs-autogen/audit_coverage.py --docs-dir docs/docs/api --quality --fail-on-quality --threshold 100
    Every new raise in a public function requires a matching Raises: entry, and every Returns: type annotation must match the function's actual return type. Adding a class to __all__ promotes its method docstrings into quality-gate scope — the gate will start enforcing them. The build-and-validate CI job enforces both with --fail-on-quality.

11. Writing Tests

See test/README.md — Authoring guide for the full authoring guide (naming, fixture discipline, mock discipline, assertion style).

  • Place tests in test/ mirroring source structure
  • Name files test_*.py (required for pydocstyle)
  • Use gh_run fixture for CI-aware tests (see test/conftest.py)
  • Mark tests checking LLM output quality with @pytest.mark.qualitative
  • If a test fails, fix the code, not the test (unless the test was wrong)
  • Static type checks live in test/typing/ as check_*.py files (not test_*.py, so pytest skips them). They use typing.assert_type inside function bodies to verify overload resolution and generic parameterization — e.g., that session.aact(..., await_result=True) narrows to ComputedModelOutputThunk[str]. Verification happens via uv run mypy .; the functions are never executed. Add a new check_*.py here when introducing or modifying @overload signatures or generic type parameters on public APIs.

12. Writing Docs

If you are modifying or creating pages under docs/docs/, follow the writing conventions in docs/CONTRIBUTING_DOCS.md. Key rules that differ from typical Markdown habits:

  • No H1 in the body — Docusaurus renders the frontmatter title automatically; a body # Heading produces a duplicate title in the published site
  • Cross-doc links: relative, with the extension — use ../concepts/requirements-system.md, not ../concepts/requirements-system, and never a root-absolute /concepts/requirements-system. With the extension Docusaurus resolves against the source file, so a rename breaks the build loudly and the link also works when browsing on GitHub. Extensionless links are raw URL paths checked only against the route table, and root-absolute ones are version-blind — from a page in docs/docs/ (the next version) they resolve to the released version's route. Numbered files keep their prefix in the link (../tutorials/02-streaming-and-async.md). The Docusaurus build does not reject extensionless links, so check with the rg sweep in docs/CONTRIBUTING_DOCS.md → Links.
  • Frontmatter required — every page needs title and description; add sidebar_label if the title is long
  • markdownlint gate — run npx markdownlint-cli2 "docs/docs/**/*.md" and fix all warnings before committing a doc page
  • Verified code only — every code example must be checked against the current mellea source; mark forward-looking content with > **Coming soon:**
  • No visible TODOs — if content is missing, open a GitHub issue instead

13. Feedback Loop

Found a bug, workaround, or pattern? Update the docs:

14. Working with Adapter Functions

Adapter functions are specialized LoRA/aLoRA adapters that add task-specific capabilities (RAG evaluation, safety checks, calibration, etc.) to Granite models. Mellea handles adapter loading and input formatting automatically — you just call the right function.

Using Adapter Functions in Mellea

Prefer the high-level wrappers in mellea/stdlib/components/intrinsic/. These handle adapter loading, context formatting, and output parsing for you:

Module Function Description
core check_certainty(context, backend) Model certainty about its last response (0–1)
core requirement_check(context, backend, requirement) Whether text meets a requirement (0–1)
core find_context_attributions(response, documents, context, backend) Sentences that influenced the response
rag check_answerability(question, documents, context, backend) Whether documents can answer a question (0–1)
rag rewrite_question(question, context, backend) Rewrite question into a retrieval query
rag clarify_query(question, documents, context, backend) Generate clarification or return "CLEAR"
rag find_citations(response, documents, context, backend) Document sentences supporting the response
rag flag_hallucinated_content(response, documents, context, backend) Flag potentially hallucinated sentences 
from mellea.backends.huggingface import LocalHFBackend
from mellea.stdlib.components import Message
from mellea.stdlib.components.intrinsic import core
from mellea.stdlib.context import ChatContext

backend = LocalHFBackend(model_id="ibm-granite/granite-4.1-3b")
context = (
    ChatContext()
    .add(Message("user", "What is the square root of 4?"))
    .add(Message("assistant", "The square root of 4 is 2."))
)
score = core.check_certainty(context, backend)

For lower-level control (custom adapters, model options), use mfuncs.act() with Intrinsic directly — see examples in docs/examples/intrinsics/.

Weights binding shapes

Adapter.weights normalizes each deployment's activation mechanism behind three shapes — a WeightsBinding lifecycle for weights you stage yourself, EmbeddedBinding.apply_activation for weights already in the served model, or ServerMediatedBinding for a model tag selected by the provider. The post-activation shape each produces:

Binding Reality Lifecycle verbs Caller invokes Normalized post-activation state
LocalFileBinding LocalFile/PEFT prepare / activate / deactivate / release activate() / deactivate(), via adapter_scope Backend-internal PEFT adapter state toggled; the outgoing request is untouched
EmbeddedBinding Embedded/Granite Switch none — weights are already in the served model apply_activation(request, identity) request.extra_body["chat_template_kwargs"]["adapter_name"] set; request.api_params["model"] removed if present
ServerMediatedBinding Ollama bundled adapter model none for the current Ollama path select the configured model tag during intrinsic generation Ollama request's model is the bundled adapter tag; full lifecycle telemetry remains follow-up work
PromptBinding Prompt fallback (weightless) prepare / activate / deactivate / release — all no-ops except activate(), which logs activate(), via the backend's weightless generate path (no adapter_scope) None — generation runs against the base model with the rewriter-injected instruction

Project Resources

  • Canonical catalog: mellea/backends/adapters/catalog.py — source of truth for adapter function names, HF repo IDs, and adapter types
  • Usage examples: docs/examples/intrinsics/ — working code for every adapter function
  • Helper functions: mellea/stdlib/components/intrinsic/rag.py and core.py

Adding New Adapter Functions

When adding support for a new adapter function (not just using an existing one), fetch its README from Hugging Face first. Each README contains the authoritative spec for input/output format, intended use, and examples.

Writing examples? The HF READMEs also document intended usage patterns and example inputs — useful reference when writing code in docs/examples/intrinsics/.

Repo Purpose Adapter functions
ibm-granite/granitelib-rag-r1.0 RAG pipeline answerability, citations, hallucination_detection, query_rewrite, query_clarification
ibm-granite/granitelib-core-r1.0 Core capabilities context-attribution, requirement-check, uncertainty
ibm-granite/granitelib-guardian-r1.0 Safety & compliance guardian-core, policy-guardrails, factuality-detection, factuality-correction

README URLs — RAG adapter functions (no model subfolder):

https://huggingface.co/ibm-granite/granitelib-rag-r1.0/blob/main/{intrinsic_name}/README.md

Core and Guardian adapter functions (include model subfolder):

https://huggingface.co/ibm-granite/granitelib-{core,guardian,rag}-r1.0/blob/main/{intrinsic_name}/granite-4.1-{3b,8b,30b}/{lora,alora}/README.md