Skip to content

feat(inference): [slice 3] bind custom hosted providers to native sandbox authority - #12889

Open
deepujain wants to merge 142 commits into
mainfrom
codex/12636-custom-hosted-fresh
Open

deepujain wants to merge 142 commits into
mainfrom
codex/12636-custom-hosted-fresh

Conversation

@deepujain

@deepujain deepujain commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Outcome

Fresh hosted OpenAI-compatible and Anthropic-compatible endpoints use native providers attached to the selected sandbox. Their validated endpoint, provider identity and credential authority survive supported lifecycle operations. Host-local endpoints and existing shared-route sandboxes retain their supported behavior.

Reason

Slice 3 requires endpoint-specific ownership through setup, authenticated requests, observation, restart, rebuild and cleanup. Missing or conflicting authority must fail without silently selecting another route or discarding recovery records.

Related issues

Fixes #12636. Part of #12558. Consumes merged #12895 for native NVIDIA credential handling and #13158 for compatible curl image packages; #13129 restores the durable E2E toolchain source.

Changes

  • Validate endpoint admission, API family, resolved addresses and immutable provider identity before granting access. Persist ownership before attachment and observe ambiguous mutations before proceeding.

  • Carry native attachments through onboarding, inference selection, status, health, export, restart, rebuild and recovery. Reconcile only the selected sandbox's endpoint policy and preserve unrelated entries. Keep upstream credentials outside agents through issued credential references, scoped HTTPS-pin or Bedrock adapter authority, and bounded projection observation.

  • Verify replacement registration before retiring superseded provider and HTTPS authority. Retain recovery records when cleanup cannot be confirmed; reject malformed or conflicting ownership before destructive mutation.

  • Use the matching provider/model session endpoint when an implicit inference switch selects its route owner. Forward the lifecycle environment to rebuilt-attachment verification. These changes are protected by OpenClaw/Hermes routing and rebuilt-smoke regressions.

  • Return to interactive endpoint selection only for credential-free URL/DNS admission failures before adapter/provider mutation; noninteractive setup exits unsuccessfully. Authority, ownership and cleanup failures remain hard failures. The existing admission table tests both modes and confirms rejection creates no provider or adapter.

  • Preserve all native live cases while separating Deep Agents Code and Docker-only Pi terminal cases from the core custom-hosted shard. The same inference-routing selector includes retained, core and terminal shards for Docker/Podman. Core limits remain 330 minutes per test and 350 per job; terminal limits are 122/142 minutes. Timeout, selector and coverage tests protect the complete group, and the terminal helper retains all live assertions while forwarding Pi’s required runtime arguments.

  • Align setup, switching and quickstart guidance with native attachments versus retained shared routes, qualifying HTTPS adapter behavior as DNS-backed HTTPS. Preserve capability audit evidence states and existing agent/API boundaries.

  • Finish later native-home removals owned by the current identity after a permission denial, then retain failure for the existing privileged/pinned-user cleanup pass. Root, protected-path and durable-identity safeguards remain enforced.

  • Preserve maintenance stop/start/readiness/abort diagnostics and add bounded failure observations for native selector timing, Hermes export policy and DCode consent/round files. These diagnostics do not establish that the underlying runtime failures are repaired.

  • Resolve an actual main conflict by retaining canonical Pi qualification receipts and their matching authority together.

Verification

Current published candidate: a6c26a7e5b4d6a7db3823d7bc358b57fb91b1e76. PR API base: 876e20ef733444b3a9c141242c84e1f16b6a7722.

The startup repair regenerates the shipped runtime, pins its exact digest, and executes it against four valid native profiles and sixteen rejection cases while retaining NVIDIA/OpenRouter exports. Pi receipt authority exactly matches its checked-in qualification receipts.

The required conflict integration preserves both the PR's native custom authority and incoming main's native local Deep Agents Code support, onboarding auto-approval interface and lifecycle behavior. The runtime bundle was regenerated from the combined source. Onboarding verification carries recorded custom/local/fixed attachments through its existing context owner. Dependency and growth limits were satisfied without raising thresholds. GitHub now reports the PR mergeable; review approval remains outstanding.

Local validation includes 530/530 final dependency-owner tests and 42/42 affected deployment-verification tests, plus the retained integration tests recorded in the delivery checkpoint. Normal isolated commit and pre-push hooks, publication builds and CLI typecheck passed. Documentation checks and the pinned Fern check passed (two warnings retained). Remote branch and PR match the signed commit; all 141 PR commits are GitHub Verified. The worktree is clean.

All 38 failed runtime logs from 3addd6aa83 were read completely and all 38 failure archives authenticated. Thirty-six first failures share the repaired startup rejection; two are inherited unsupported Podman custom-Dockerfile fixtures. Downstream assertions and prior failures remain separately dispositioned. This does not establish that all 38 runtime jobs now pass.

Check Current candidate
CI Running.
Managed Images Running; collect the entire matching-head qualification.
Docs Preview/build and changed links passed; CLI/reference parity collection continues.
E2E — automatic Portable Running; one main-only job skipped.
E2E — automatic Podman CPU Running.
E2E — selected staging Dispatched; builds its own exact-head image.
E2E — selected native Not dispatched; six runtime jobs retained, awaiting matching-head managed-image qualification.
E2E — selected regressions Not dispatched; original 23 job selectors and two targets, 58 runtime jobs retained, awaiting matching-head images.
PR Advisor Current review running. Previous 0857dac8a4 run had nine clear bounded reports, not full-diff hosted coverage. Prior challenged findings are not claimed withdrawn.
CodeRabbit Limited historical coverage; paused automatic review is not a full current-head review.
CodeQL / human review Current scanning is running. Six historical annotations have no waiver; human changes requested remains.

Prior selected runtime results on 3addd6aa83: staging 1/1 passed; native 4/6 passed, 2 failed; regressions 22/58 passed, 36 failed. Setup and aggregate jobs are excluded. All runs are terminal. Retain missing original Hermes export-time policy, native Podman HTTP 530 origin, hosted authorization cleanup permissions and survival final-absence evidence; no speculative repairs or weakening of security policy. The two unsupported Podman fixtures remain excluded from source repairs, not waived.

The complete prior diff and current repair delta have specialist-based self-review, not independent approval or a full-diff hosted Advisor pass. Preserve all source/assertion ledgers, failure history, retry counts and the separately pinned terminal VM.

Review notes

Sensitive paths include inference, onboarding, credential custody, ownership persistence, sandbox lifecycle and E2E enforcement. Current-head independent review, original six-task acceptance and all selected runtime qualification remain outstanding. No merge authorization.

The latest batch reconciles the census metadata to the actual unchanged live assertions; it does not raise an enforcement limit or edit the approved exception. The exact assertion transition approval remains resolved: maintainer record, digest 92e6d8923acbda85e9d817f68145ccd260a5f7be0e33ec87af201079d1323c2f. Per-case deadlines remain unchanged. The retained native routing job includes incoming main cases and uses their combined calculated 348-minute budget. Credential scope, isolation rules and product scope are preserved.


Signed-off-by: Deepak Jain deepujain@gmail.com
Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

Summary by CodeRabbit

  • New Features
    • Connect hosted custom inference endpoints directly to individual sandboxes, with credentials kept outside the sandbox and scoped to the selected endpoint.
    • Use custom hosted inference across OpenClaw, Hermes, Deep Agents Code, and Pi, with route selection, status checks, and readiness verification.
    • NVIDIA Endpoints and supported fixed hosted providers use sandbox-attached providers; native local Deep Agents Code support is preserved alongside retained shared-route behavior.
    • Custom routes and credentials persist across restarts and rebuilds, with cleanup limited to resources whose ownership can be verified.
    • Expanded end-to-end coverage for custom hosted inference and native terminal clients.
  • Documentation
    • Updated setup, routing, security, recovery, and provider guidance for custom endpoints and shared routes.

@deepujain deepujain self-assigned this Oct 9, 2026
@deepujain

Copy link
Copy Markdown
Collaborator Author

NemoClaw-E2E-Growth: approve b56f349dea062d8a839d232d89919b1a120f9727e8404d61a1717a24befa3f3a

…rity

Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 9, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Fresh hosted custom endpoints now use providers attached to the selected sandbox. The changes add endpoint admission, scoped credential references, persisted provider authority, and transport handling. Onboarding, inference switching, status checks, rebuilds, resets, and destruction now carry or validate custom attachment authority.

Changes: Native custom inference

Layer / File(s) Summary
Endpoint profiles and transport
src/lib/inference/native-custom/*, src/lib/inference/https-pin-runtime*
Custom profiles bind supported APIs, admitted endpoint addresses, executable rules, and credential environments. DNS-backed HTTPS and classified Bedrock endpoints use their respective adapters.
Onboarding and inference switching
src/lib/onboard/setup-inference.ts, src/lib/actions/inference-set*, src/lib/onboard/sandbox-create/*
Onboarding prepares and attaches native custom providers. Inference switching updates sandbox policy and configuration, and handles departures, recovery, and provider retirement.
Persisted authority and sandbox lifecycle
src/lib/state/registry*, src/lib/actions/sandbox/*, src/lib/onboard/resume/*
Registry and session data validate custom attachments. Start, status, doctor, rebuild, reset, and destroy paths verify or clean up provider authority.
Agent configuration and credential references
agents/hermes/*, agents/langchain-deepagents-code/*, agents/pi/*, src/lib/onboard/managed-startup/*
Agent configuration selects compatible-provider credentials and uses scoped references for attached routes.
Validation and E2E coverage
test/*, tools/e2e/*, ci/*, docs/*
Tests and E2E targets cover profile admission, authority validation, routing, recovery, and cleanup. Documentation describes the routing paths and endpoint constraints.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Onboarding
  participant SetupInference
  participant NativeCustomProvider
  participant AuthorityRegistry
  participant Sandbox
  Onboarding->>SetupInference: Submit endpoint, API, and credential
  SetupInference->>NativeCustomProvider: Prepare profile and attach provider
  SetupInference->>AuthorityRegistry: Persist attachment authority
  SetupInference->>Sandbox: Probe selected inference route
  Sandbox-->>SetupInference: Return inference result
  SetupInference-->>Onboarding: Return selected attachment
Loading
Pre-merge checks: Passed 3 · Failed 1 · Inconclusive 1

Failed Failed checks: 1 warning · 1 inconclusive

Check name Status Explanation Resolution
Linked Issues check Warning [#12636] The implementation and focused tests cover native endpoint admission, endpoint-specific provider ownership, credential references, lifecycle propagation, recovery, isolation, and cleanup. The… Restore the tested workflow toolchain and dispatch all six current-head Docker and Podman retained, core, and terminal jobs. Run the complete supported regression selection. Record passing evidence for endpoint admission, selected-model inf…
Docstring Coverage Inconclusive Docstring coverage is 20.75% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 106 functions across 52 files. (239 skipp… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 3 passed
Check name Status Explanation
Out of Scope Changes check Passed The changes remain connected to [#12636]. Native routing, authority persistence, credential custody, lifecycle recovery, cleanup, documentation, focused tests, image qualification, and E2E selector an…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: binding custom hosted providers to native sandbox authority.
Full details: Linked Issues check

Explanation

[#12636] The implementation and focused tests cover native endpoint admission, endpoint-specific provider ownership, credential references, lifecycle propagation, recovery, isolation, and cleanup. The required current-head qualification remains incomplete. The PR states that all six native inference-routing jobs and selected regressions were not dispatched because the workflow references a deleted artifact. Earlier runtime results passed only three of six jobs and failed on OpenClaw credential references and Pi runtime arguments. Therefore, current-head evidence does not establish A1–A6, including fresh inference, restart recovery, isolation, cleanup, and no shared-route fallback.

Resolution

Restore the tested workflow toolchain and dispatch all six current-head Docker and Podman retained, core, and terminal jobs. Run the complete supported regression selection. Record passing evidence for endpoint admission, selected-model inference and observation, credential safety, restart and rebuild recovery, sandbox isolation, ownership-safe cleanup, and hard failure without shared-route fallback.

Full details: Docstring Coverage

Explanation

Docstring coverage is 20.75% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 106 functions across 52 files. (239 skipped: 44 unsupported, 195 over the file limit.)

✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: TypeScript

TypeScript / code-coverage/plugin

main 63002cd codex/12636-custom-h... a6c26a7 +/-
96% 97% +1%
View coverage of the 10 most impacted files.
File main 63002cd codex/12636-custom-h... a6c26a7 +/-
nemoclaw/src/onboard/config.ts 98% 96% -2%
nemoclaw/src/index.ts 94% 93% -1%
nemoclaw/src/bl...print/runner.ts 95% 95% 0%
nemoclaw/src/bl...ime-identity.ts 97% 97% 0%
nemoclaw/src/bl...t-management.ts 100% 100% 0%
nemoclaw/src/co.../config-show.ts 100% 100% 0%
nemoclaw/src/commands/slash.ts 100% 100% 0%
nemoclaw/src/ru...time-context.ts 100% 100% 0%
nemoclaw/src/sh...er-boundary.cts 100% 100% 0%
nemoclaw/src/on...native-route.ts 0% 100% +100%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated October 16, 2026 06:39 UTC

Comment thread src/lib/inference/https-pin-runtime.ts Dismissed
Comment thread src/lib/inference/native-custom/profile.ts Dismissed
Comment thread src/lib/inference/native-custom/profile.ts Dismissed
Comment thread test/e2e/support/https-pin-compatible-server.test.ts Fixed
Comment thread test/e2e/support/https-pin-compatible-server.test.ts Fixed
@deepujain

Copy link
Copy Markdown
Collaborator Author

NemoClaw-E2E-Growth: approve 1360e1dc6a23f4601f8ab14eebfb1d4d9a54854de95623a6007a73f6bc622174

@deepujain

Copy link
Copy Markdown
Collaborator Author

NemoClaw-E2E-Growth: approve 647470d10d59d3fe7716d3e96c48ca5253d411b68d6bcf6a11e25ca18e029dd5

Signed-off-by: Deepak Jain <deepujain@gmail.com>
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

@deepujain

Copy link
Copy Markdown
Collaborator Author

NemoClaw-E2E-Growth: approve 21603c26db9785e242ee55b99eb4b818b509c5e7666ccdf6116fe315f46e8b69

@deepujain

Copy link
Copy Markdown
Collaborator Author

Qualification scope: deterministic selection, credential-reference, identity, and fixture behavior are covered locally. Live qualification is limited to the real OpenShell boundary: fresh native custom onboarding, a fresh agent request through the attached provider, restart authority, peer isolation, and owned-resource cleanup. The existing inference-routing target now includes native Anthropic Messages; host-local Anthropic passes do not establish that path.

Optional Brev access was checked using the existing running instance. CLI login/listing succeeded, but SSH failed with public-key authentication after one supported refresh and retry. No remote test ran and no instance was created or changed. This external limitation does not stop the GitHub qualification path. Native Podman qualification separately encounters unavailable artifacts at its trusted toolchain pin; its earlier failure occurred before candidate execution.

The onboarding repair and actual main-conflict integration are committed locally. The remaining Anthropic evidence repair is being validated as one batch before publication. Final-candidate runtime evidence and Pi image receipts remain pending; this PR is still draft.

@deepujain

Copy link
Copy Markdown
Collaborator Author

NemoClaw-E2E-Growth: approve 0f29b8054e04df3d5614865b4b363f43f78d209d0ca7d967f1779bb2c5db01e7

@deepujain

Copy link
Copy Markdown
Collaborator Author

Native runtime qualification remains pending. I split the unchanged TC-INF-11 lifecycle into a bounded 250-minute target because the original file allowed 390 minutes of serial tests before cleanup. All cases and assertions are preserved; the retained routing job now has a 180-minute bound.

The trusted main planner does not yet select the new native target, and still gives the retained routing job 75 minutes. I will avoid unsupported or knowingly undersized dispatches and continue applicable checks. This limitation does not establish native restart or credential-resolution success.

Brev SSH and the pinned Podman toolchain limitations are already recorded above. No remote Brev test ran. Manual execution is justified only for real OpenShell attachment, credential resolution, agent requests, restart, isolation and cleanup; deterministic catalog/configuration behavior is covered locally.

@deepujain

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Too many files to review.

Comment on lines 102 to +117
normalizedPinoRedactFixture
.replace(/(?:Bearer\s+)?openshell:resolve:env:[A-Za-z0-9_]+/giu, "unused")
.replace(/sk-OPENSHELL-RESOLVE-ENV-[A-Za-z0-9_-]+/gu, (candidate) =>
isSafeCredentialPlaceholder(candidate) ? "unused" : candidate,
)
.replace(/(?:xox[bx]|xapp)-OPENSHELL-RESOLVE-ENV-[A-Za-z0-9_-]+/gu, (candidate) =>
isSafeCredentialPlaceholder(candidate) ? "unused" : candidate,
)
// Generated bundles contain the accepted-placeholder matcher itself.
// Normalize that exact source fragment without stripping the reserved
// prefix from malformed placeholder-shaped credential values.
.replace(/(?:xox[bx]|xapp)-OPENSHELL-RESOLVE-ENV-\[A-Za-z0-9_\]\+/gu, "unused")
.replaceAll(
"/^sk-OPENSHELL-RESOLVE-ENV-(?:(?:v[0-9]{1,20}|s[a-f0-9]{64})_)?COMPATIBLE(?:_ANTHROPIC)?_API_KEY$/",
"unused",
)
Comment on lines +93 to +107
const result = spawnSync(
"bash",
[
"-c",
'source "$1"; assert_identity "$2" test-model initial "$3"',
"identity-check",
dcodeFreshReonboardCheck,
identity,
expected,
],
{
encoding: "utf8",
env: { PATH: DEFAULT_TEST_PATH },
},
);
@deepujain

Copy link
Copy Markdown
Collaborator Author

Advisor finding F-reduction-simplification-3b4908528925beffc4d1 from run 38661071123 attributes DGX Spark qualification to this PR. The current base comparison does not support that attribution:

  • Head: ba81e3de6f15009319038f14892370bbf5a74b89; base: c6c3fe268ff19f77b1abedca821a0f70973f43f6.
  • .github/workflows/e2e.yaml has the identical Git blob bd2aaafa20958b74be489fe80efb5b505c2d4488 at both revisions. It is absent from this PR's current base-to-head diff.
  • The Spark workflow and implementation came from merged fix(inference): repair llama.cpp qualification contracts #13114 (ba589b1da6d6e4ea99b21e16cc5fc0f97b97cf5b), which is already an ancestor of the base. Its qualification manifest, runner, plan export, receipt validation, and protocol code are also byte-identical between base and head.
  • The two overlapping files with PR differences (test/e2e/README.md and test/e2e/support/workflow-plan.test.ts) add native custom inference documentation, two inference targets, and runtime metadata validation; their current diff adds no Spark behavior.

The bounded review compared an older reviewed commit with this head and included merged upstream work. Please reassess the scope finding against the current base; if this evidence invalidates it, withdraw it, otherwise identify the candidate-owned Spark change. The associated Spark E2E recommendations have the same attribution issue. I am preserving the upstream workflow and the agreed Slice3 runtime scope. The Advisor gate remains failed; this response does not waive it or claim merge readiness.

@github-actions

Copy link
Copy Markdown
Contributor

PR Review Advisor finished for commit d45cd94. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

Request review only when Require no Advisor blockers is green.

All previous runs

@deepujain

Copy link
Copy Markdown
Collaborator Author

NemoClaw-E2E-Growth: approve 92e6d8923acbda85e9d817f68145ccd260a5f7be0e33ec87af201079d1323c2f

@github-actions

Copy link
Copy Markdown
Contributor

PR Review Advisor finished for commit f1ce907. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

Request review only when Require no Advisor blockers is green.

All previous runs

@github-actions

Copy link
Copy Markdown
Contributor

PR Review Advisor finished for commit 9226aa9. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

Request review only when Require no Advisor blockers is green.

All previous runs

deepujain and others added 4 commits October 15, 2026 09:02
@github-actions

Copy link
Copy Markdown
Contributor

PR Review Advisor finished for commit 1cce637. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

Request review only when Require no Advisor blockers is green.

All previous runs

@deepujain

Copy link
Copy Markdown
Collaborator Author

Advisor follow-up for 1cce637a49a41d470bf213dc71d5247c396d38e1 against 150284c3470d2fffba7f380b84ac9ffe553169b7 (run 38783767112): all nine specialist artifacts were authenticated and inspected. Seven are clear; the two P1 findings below need correction or concrete counterevidence.

Delivery — F-delivery-flow-1aad0af551d6b58cafc2. The proposed overwrite mechanism conflates a GitHub Actions check run with a commit status. The installer workflow creates check run 116580802825; admission creates a separate commit status under the same name (run 38780980836, status ID 56453878665). Both records coexist on this head. GitHub documents that when a check and commit status have the same required name, both must pass: required-check documentation. Both workflow files are byte-identical to the current PR base. Please withdraw this overwrite finding, or identify a second commit-status producer or a concrete consumer that accepts admission success while the installer check fails. Renaming the admission status without checking its required-gate consumer could change the existing enforcement contract.

Documentation — F-documentation-standard-work-e98b4c2d8962bc1f9b2c. The cited recovery command at line 468 immediately follows “If the restart fails, the route and configuration remain committed.” The same section then explicitly says to run the printed command if the gateway restart fails after configuration is committed. completeInferencePostCommit implements that failure-only recovery path. Please withdraw the P1 claim that this instructs an unconditional second restart, or identify the rendering/context that removes that condition. Repeating the condition in the next sentence may be a wording preference; the recorded failure condition already matches the behavior.

No findings have been self-waived or marked resolved. The Advisor gate remains failed pending corrected review or disposition. The candidate is unchanged while existing image activation and selected staging E2E runs finish; the existing selected E2E plan is preserved.

@github-actions

Copy link
Copy Markdown
Contributor

PR Review Advisor finished for commit d24cea9. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

Request review only when Require no Advisor blockers is green.

All previous runs

@deepujain

Copy link
Copy Markdown
Collaborator Author

Evidence-based challenge to the two P1 findings in Advisor run 38824308337, for head 3addd6aa83d2526536f7a0a17bf884cb087da1f4:

  1. F-verification-mistake-proofing-c6574326111d250f2306 — dependency preparation. The actual PR workflow caller checks out the PR base into .trusted-ci-actions and invokes that copy of ci-cli-coverage-merge. The dependency script derives trusted_root from its own location, executes preparation from that root, then unsets GitHub tokens before candidate npm installation with --ignore-scripts (source). This contradicts the stated premise that the preparation module executes from the candidate checkout. Please withdraw this finding or identify the concrete candidate-controlled executable path that crosses that boundary.

  2. F-architecture-standard-work-90d7f42326b154a22aca — status overwrite. Admission writes a legacy commit status using createCommitStatus (source); installer validation creates an Actions job/check-run named check-hash (source). They are distinct records. GitHub documents that a check and commit status sharing a required name must both pass. Please withdraw the claimed overwrite/bypass or provide a reproduction under the actual repository rule. Naming clarity alone does not establish the reported P1 failure.

The affected owners and callers are byte-identical across the previous head, this head, and its exact base. That comparison is attribution evidence, not by itself a security proof. These are disputed findings awaiting resolution; this response does not waive either gate or claim approval.

Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
prekshivyas added a commit that referenced this pull request Oct 16, 2026
…3201)

## Outcome

Fresh custom hosted endpoint selections use an endpoint-specific native
provider attached to the selected sandbox. Agent configuration,
inference get, probes, and health retain the selected endpoint and API.
Native failures do not fall back to the shared route.

## Reason

This implements the bounded retry accepted in #13195, using the existing
native-provider lifecycle and OpenShell 0.0.116.

### Related issues

Fixes #13195. Part of #12558. Refs #12636 and #12889 as historical scope
and review evidence; their closure is outside this PR.

## Changes

- Bind a generated profile to the validated endpoint, API, credential
source, and admitted IPs. Existing checked-in profiles supply the
executable and credential boundary. Custom selections cannot safely
repoint a public vendor profile. `custom.test.ts` protects profile
collisions, SSRF admission, API paths, credential custody, and ambiguous
creation.
- Reuse verified provider authority without credential rotation. Missing
authority and credentials stop before provider mutation. Preserve the
existing Bedrock host adapter and legacy HTTPS-pin routes.
- Connect custom receipts to existing onboarding, switching, registry
ownership, rebuild preflight, configuration, health, and cleanup.
Switching removes conflicting access before attachment and before
rollback restoration. Tests cover selected-sandbox isolation and
retained recovery authority.
- Keep existing shared-route selections on their current path.
Unavailable legacy routes receive conditional recreation guidance
without automatic migration or an asserted cause.
- Extend the existing full E2E fixtures to inspect custom provider
identity and attachment and retain the native endpoint across restart.
The real boundary remains OpenShell credential injection, sandbox
networking, agent inference, and restart. Existing live outcome
assertions remain; deterministic support tests own route selection and
command construction. Bedrock retains its existing live owner. The
existing Anthropic switch selector now exercises an authenticated native
destination after its native authenticated baseline lifecycle. No new
live target or workflow is added.
- Extract the existing endpoint normalization functions unchanged so the
new consumers fit the repository dependency budgets. Existing imports
remain compatible.

## Verification

### Native export endpoint repair (`71a7901`)

- Fix the reproduced export rejection from the review: managed native
profiles validate the selected endpoint and runtime endpoint against
their separate verified receipt fields. Anthropic OpenAI `/v1`
translation and the Bedrock adapter remain valid; endpoint drift remains
rejected.
- All three added positive cases failed before the repair with `Registry
and live inference endpoints differ.` Final validation passed 298 tests
across `live-export-source.test.ts` and `verify-export-source.test.ts`,
including selected/runtime/both-endpoint drift controls. Normal
signed-commit hooks passed without increasing budgets. No secret values
were read by the regression cases; the diff contains no secrets.
- Prior candidate `afea1a7c05f3383bd20d5f79a477c0c8c0179423` completed
its PR check rollup with 87 successful and 40 skipped checks, including
managed images and GPU qualification. All nine [Advisor specialist
reports](https://github.com/NVIDIA/NemoClaw/actions/runs/38858389798)
were collected: no P0 and two P1 findings (Hermes native custom switch
coverage and public route documentation). Their [recorded
deferrals](#13201 (comment))
remain; the separate Advisor blocker gate failed. Prior CI evidence does
not establish a pass for this repair commit.
- Current-commit CI and scheduled automated reviews are pending. The
earlier request-changes review remains for independent reassessment; no
approval or merge readiness is claimed.


### Anthropic source selection and native-route merge repair

- Previous candidate `afea1a7c05f3383bd20d5f79a477c0c8c0179423` contains
selector repair `a472ac7` and one merge of canonical main
`a304c5bd67fb0badff1fd5b256dc09d1a1268e1c`. The existing Anthropic
switch target now selects changes to its native-provider, onboarding and
native-selection owners. Five omitted paths failed before repair; the
final boundary suite passed 14 tests. Existing planner coverage passed
136 tests before relocation; these results overlap the earlier combined
150-test run.
- Resolve the required main conflicts while preserving custom-hosted
endpoint/API/credential authority and main's native-local routing.
Integration regressions exposed dropped custom binding and endpoint
arguments, TLS readback, and native-local interpretation; these were
repaired in their current owners. The final affected
route/configuration/recovery batch passed 228 tests in six suites. The
onboarding handler passed 38 tests after moving its unchanged
native-route exclusion into the existing refresh predicate. Earlier
failed integration runs and their diagnoses remain retained. These runs
overlap; no summed test total is claimed.
- The assertion census passed with 1,272 direct expectations across 77
live files. Strict Pi receipt/source parity passed; the existing paired
receipts remain qualified. Original signed-commit and publication hooks
passed, including final CLI type checks, without increasing architecture
or complexity limits. The diff contains no secrets.
- Prior candidate `53bc898` completed evaluation before replacement. Its
[CI
retry](https://github.com/NVIDIA/NemoClaw/actions/runs/38843616981/attempts/2)
passed after one explicitly requested retry. The original 5,313 ms test
timeout at an unchanged 5,000 ms deadline remains retained with
undetermined cause; the passing retry does not establish infrastructure
failure. [Managed
images](https://github.com/NVIDIA/NemoClaw/actions/runs/38843616977) and
[GPU
qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/38843618739/job/116799363127)
passed with authenticated candidate and cleanup evidence. CodeRabbit and
both scheduled sets of nine Advisor reports completed. These results are
historical for the current candidate.
- The source-selection finding is repaired here. The later [security
review](https://github.com/NVIDIA/NemoClaw/actions/runs/38850077726)
identified retained access after removing an admission allowlist entry.
The accepted policy preserves verified durable private bindings:
unsetting `NEMOCLAW_TRUSTED_PRIVATE_HOSTS` does not revoke an existing
binding. Fresh or changed private endpoints still require admission;
reuse retains exact endpoint/API/provider identity and admitted IPs.
This policy disposition does not establish arbitrary receipt safety or
live private-endpoint coverage. Public documentation remains with the
accepted post-merge owner.
- Current-candidate CI, CodeRabbit, all nine Advisor reports, images/GPU
qualification and authorized manual live validation remain pending. The
approved live scope is 21 existing manual jobs plus staging Launchable.
Protected runtime remains held for the separate timeout follow-up and
cleanup reconciliation; no retry allowance, timeout increase or all-pass
waiver is introduced.
- Three combinations remain documented follow-ups: Hermes native custom
Anthropic-to-OpenAI with a real request, fresh public custom OpenAI
onboarding, and Pi native credential rewrite followed by a real request.
The existing Hermes switch, custom endpoint fixtures and Pi
qualification are the nearest coverage, but do not prove those
combinations. Extending their contracts is deferred to follow-up work;
this PR does not claim that evidence.

### Historical resume credential handoff and merge resolution
(`53bc898`)

- Published candidate `53bc89879a90e533a191f8b8df2022dcad092ab6`
integrates canonical main `b5efa876a775265b63ee963f0b363cc650bdd87e`.
The sole conflict was the E2E assertion census. Resolution preserves
this PR's existing probe increase and main's protected-runtime count
reduction. The census check passed with 1,260 direct expectations across
77 live files.
- Fix the single resume defect reported by the migration and operability
Advisor reports. When the session has no credential name, resume obtains
it from the validated custom attachment for the selected provider and
normalized endpoint. Explicit selections take precedence. Actual setup
still checks API and live provider identity before reusing authority.
Reservation and legacy paths retain their existing behavior.
- Two regression tests failed before the fix: the shim returned the
default name instead of `TEAM_KEY`, and actual setup required a new
credential. Final focused validation passed 58 tests across
`resume-provider-shim.test.ts`, `resume-provider-recovery.test.ts`, and
`native-provider/custom.test.ts`. Controls cover explicit selections,
both custom provider kinds, endpoint/provider mismatch, invalid
attachment, normalized endpoint and reuse without credential rotation.
- Strict Pi receipt validation and source parity passed for this commit;
the existing paired receipts remain valid. Original signed-commit hooks
passed. A local dependency-limit failure was repaired by placing receipt
lookup in the existing custom-provider owner, without changing budgets.
The standalone compiler passed with the existing 8 GiB hook allowance
after its default 4 GiB run exhausted the heap. Original publication
validation and final CLI type checks passed. The diff contains no
secrets.
- Evaluation of `b8065df` completed before this repair. [Primary
CI](https://github.com/NVIDIA/NemoClaw/actions/runs/38835151923) passed
23 jobs with two legitimate skips. [Managed
images](https://github.com/NVIDIA/NemoClaw/actions/runs/38835151800)
passed 14 jobs with four skips; AMD64, ARM64 and rootless activation
each passed 13 phases and 13 cleanup actions. [GPU
qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/38835153672/job/116774805269)
passed device approval, cron execution, the later agent turn, all six
phases and all three cleanup actions. Podman CPU and portable rootless
qualification passed. These results are historical for the new commit.
- CodeRabbit and all nine [Advisor reports for
`b8065df`](https://github.com/NVIDIA/NemoClaw/actions/runs/38836956611)
completed. Two Advisor findings share the resume defect repaired here;
public documentation retains its authorized post-merge owner. The
CodeRabbit suggestion to split or remove Pi receipts is a false
positive: the unchanged contributor contract requires this follow-up in
the same PR, and strict source parity passed. The repeated Pi
credential-rewrite request remains an unverified combination.
- New-commit CI, automated review, image and GPU qualification remain
pending. The 23 selected manual E2E jobs have no result for this commit.
Retained Hermes, fresh public custom OpenAI and Pi combination gaps
remain open. The protected-runtime timeout follow-up remains outside
this PR, with NIM, rollback and final fixture cleanup unverified. That
scheduled cycle completed before the selector repair.

### Historical Pi qualification receipt follow-up (`b8065df`)

- Published candidate `b8065df5b1de956b533c278b77b8ba23a3efba5e` changes
only the AMD64/ARM64 Pi qualification receipts and their matching
authority hashes. The exact contract bytes came from [successful
managed-image run 38826692761, attempt
1](https://github.com/NVIDIA/NemoClaw/actions/runs/38826692761), source
`19f521416a5dbb62ac9fa169e5f03eb817c72297`, cohort
`ghrun-38826692761-1`. Artifact identity, archive size and SHA-256,
source, platform and common run/attempt were authenticated. The
authority hashes are computed from the receipt bytes.
- Strict Pi receipt validation and source parity passed. Qualified Pi
image inputs are unchanged by this three-file update. Normal
signed-commit and publication hooks passed with original hook execution
inside the verified credential boundary. No new behavior or test was
added; the strict receipt/source-parity check verifies this data update.
The diff contains no secrets.
- Evaluation of source `19f5214` completed: image publication passed 14
jobs with four legitimate skips; AMD64, ARM64 and rootless activation
each passed 13 phases and 13 cleanup actions. [GPU
qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/38826694610/job/116748692376)
passed explicit device approval, cron execution, a later agent turn, all
six phases and all three fixture cleanup actions. The original
device-denial cause remains unresolved. These are results for the
qualified source, with independent new-commit checks still pending.
- CodeRabbit completed review of `19f5214`, resolved both routing
findings and reported no new actionable finding. All nine [Advisor
reports](https://github.com/NVIDIA/NemoClaw/actions/runs/38829342333)
completed. The aggregate retained the known Hermes combination gap and
public-documentation findings, with prior source-backed and authorized
post-merge dispositions. Primary CI failed solely the stale Pi receipt
gate addressed here and its aggregate; all 12 CLI shards passed.
- Evaluation of `b8065df` completed as summarized above. Applicable
manual live E2E evidence for that commit remains unavailable. Historical
ordinary E2E passes do not establish its result. Hermes native
Anthropic-to-OpenAI, fresh public custom OpenAI onboarding and Pi
credential-rewrite combination gaps remain recorded. The
protected-runtime timeout stays outside this PR, with NIM, rollback and
final fixture cleanup unverified.

### Historical routing API repair (`19f5214`)

- Published candidate `19f521416a5dbb62ac9fa169e5f03eb817c72297` fixes
the two reproduced custom Anthropic API-consistency findings. Native
binding, persisted API selection and agent configuration share one
resolved default; explicit native Responses keeps its request path and
inference namespace. Fixed-provider identity checks and existing
shared-route behavior remain covered.
- Focused validation: `vitest run --project cli
src/lib/inference/native-provider/custom.test.ts
src/lib/onboard/setup-native-hosted.test.ts
src/lib/sandbox/compose-sandbox-config-body.test.ts
src/lib/inference/config.test.ts` passed 187 tests across four files.
After consolidating the API resolver, its three affected suites passed
136 tests, overlapping that total. Normal signed-commit and publication
hooks passed without increased complexity, architecture or growth
limits. The diff contains no secrets.
- Evaluation of `19f5214` completed as summarized above. The genuine
source-matching paired Pi receipts are applied in the receipt-only
follow-up. The earlier `94c8ec0` receipts remain historical and were not
applied to changed routing source.
- [GPU qualification on preceding candidate
`94c8ec0`](https://github.com/NVIDIA/NemoClaw/actions/runs/38817587355/job/116720057849)
passed explicit device approval, cron execution, a subsequent agent turn
and all three fixture cleanup actions. The old denial did not recur; its
precise cause remains unresolved. This result does not certify the new
routing commit.
- The protected-runtime timeout remains the recorded infrastructure
fast-follow outside this PR. NIM, rollback and final fixture cleanup
remain unverified; no all-pass result or broad waiver is claimed.

### Historical merge repair (`94c8ec0`)

- Published candidate `94c8ec0b0dd0ada2161719bb082cab0bb693f739` merges
canonical main `d1c1843147ba40a7d67c1a48affbba7c899010d9`. It preserves
custom-native OpenAI/Anthropic behavior, fixed NVIDIA/OpenRouter
identity and endpoint checks, and main's atomic credential-handle
refresh.
- The initial local repair passed 221 tests. Commit hooks then
identified two dependency-limit violations; sharing the existing
API-aware provider-key mapping removed both added imports without
increasing limits. Four affected configuration and materialization
suites passed 184 tests. After integrating the overlapping main fix,
four startup/refresh suites passed 217 tests. These runs overlap and are
not additive.
- Mapping checks: `npx --no-install vitest run --project cli
src/lib/sandbox/compose-sandbox-config-body.test.ts
src/lib/inference/config.test.ts
src/lib/inference/native-provider/custom.test.ts --project integration
test/generation/generate-native-hosted-config.test.ts`. Startup checks:
`npx --no-install vitest run --project integration
test/agents/openclaw/runtime/provider-placeholder-refresh.test.ts
test/agents/openclaw/runtime/nemoclaw-start.test.ts
test/mcp/mcp-tool-discovery-image-contract.test.ts --project cli
src/lib/sandbox/compose-sandbox-config-body.test.ts`.
Source-architecture checks passed without increased budgets or new
cycles.
- Normal commit and publication hooks passed inside a verified
credential-isolation boundary. Original hook scripts, arguments, input
and failure status were preserved. Git transport authentication stayed
outside the hook boundary. The diff contains no secrets.
- Evaluation of `94c8ec0` completed. Images passed all 14 executed jobs,
GPU qualification passed, and CodeRabbit plus all nine Advisor reports
were collected. Primary CI failed the strict Pi receipt-refresh gate.
CodeRabbit identified the two API-consistency defects repaired above.
Advisor public-documentation and Pi combination-gap findings retain
their recorded dispositions. These results are historical for the new
routing commit.
- Infrastructure fast-follow outside this PR: align the
protected-runtime job, test and cleanup budgets and bound NIM
acquisition. On `8e004`, GitHub's 75-minute job limit expired after all
three agents passed Ollama/vLLM; NIM, rollback and final fixture cleanup
remain unverified. Reconcile run-owned cleanup before replay. This
remains incomplete qualification.

### Historical cycle 6 candidate

- Integrated canonical main `f2ed1705ce2270efd4a3dda0b7ea09ea9efacd63`
in `8fb87cba9a7864465c9deb9423318766af7f1239`, preserving the PR's
source and test changes. Nine focused suites passed 263 tests.
- Final candidate `8e004b2ecee0c34d8fafc9a41868fcadf693fb24` changes
only the AMD64/ARM64 Pi qualification receipts and their matching
authority digests. The trusted main collector authenticated both
artifacts from [successful image publication
38770328038](https://github.com/NVIDIA/NemoClaw/actions/runs/38770328038),
including digest, size, source revision, architecture and common
run/attempt. That publication passed all-agent activation on AMD64,
ARM64 and rootless Podman.
- Strict Pi source parity reports `qualified`. Focused
receipt/source-parity and candidate-authority suites passed 34 tests.
Normal commit and publication hooks passed; no hook bypass or force
push. The receipt follow-up is the final authorized publication in cycle
6.
- Integrated-candidate CodeRabbit and all nine [Advisor attempt 2
specialist
reports](https://github.com/NVIDIA/NemoClaw/actions/runs/38773784513/attempts/2)
are complete. The one infrastructure retry recovered the missing
migration review. Advisor's aggregate still reports the previously
dispositioned Hermes combination gap and public-documentation drift;
these retain the [existing
dispositions](#13201 (comment)).
The reported missing native Anthropic selector resolves to the existing
`openclaw-inference-switch` job, whose Anthropic shard selects native
custom mode.
- The fresh public custom OpenAI onboarding combination remains
explicitly unverified in live E2E, alongside the existing Hermes and Pi
combination gaps. Retained deterministic and shell-request evidence
covers constituent behavior; it does not establish those full
combinations. No behavior failure was observed for these gaps, and no
additional scenario is claimed as passed.
- Evaluation of `8e004` completed: primary CI and images passed,
CodeRabbit and all nine Advisor reports were collected and
dispositioned, and all 26 ordinary runtime jobs acquired passing
evidence across the retained original and diagnostic attempts.
Twenty-two of 23 selected E2E selectors passed. Protected runtime timed
out with NIM/rollback/final fixture cleanup unverified; separate GPU
device approval failed. Original HTTP and dependency failures remain
recorded, with their causes unresolved. These results do not establish
the new merge commit's outcome.

### Cycle 5

- Candidate `9029658852ce5600bbfcfae943be13e265a482cf` repairs native
provider handle materialization and refresh across the existing
inference, OpenAI and Anthropic config fields. Raw, missing, unscoped
and mismatched handles remain rejected or unchanged at their owning
boundary. NVIDIA endpoint validation and the existing bounded
pre-mutation wait remain intact.
- Corrected Hermes export fixture category handling and custom HTTPS
export mock parity. Refusal still requires both aliases to exit 2,
matching diagnostics, no output files, and no credential leakage.
Refreshed both Pi receipts and authority digests from [successful image
run
38707069678](https://github.com/NVIDIA/NemoClaw/actions/runs/38707069678).
Corrected the owning E2E README baseline description.
- Local checks: native config command tests 29 passed; native export
tests 8 passed; startup refresh and Hermes fixture tests 71 passed. CLI
type checking with a 12 GiB heap and strict Pi receipt validation
passed. Both original HTTP and HTTPS smoke reproductions now pass. Tests
caught and verified the repair of an intermediate NVIDIA placeholder
regression.
- Completed the affected source and caller review, retaining the 563-row
channel, 114-row native and 984-row messaging ledgers. No live target,
retry, deadline or assertion-budget increase. The diff contains no
secrets; test values are synthetic.
- Cycle 4 [standard E2E
38712222640](https://github.com/NVIDIA/NemoClaw/actions/runs/38712222640)
failed with the native and Hermes fixture defects repaired here. Its
protected-runtime job failed with authenticated runner disk exhaustion;
logs, runtime results and cleanup evidence are unavailable, so that
result remains inconclusive. No unchanged workflow retry or base replay
was dispatched. Separate [Launchable
38712268470](https://github.com/NVIDIA/NemoClaw/actions/runs/38712268470)
passed with source/image identity and workspace absence verified.
- Normal commit and publication hooks passed, including CLI types,
repository/growth checks and secret scanning. GitHub verified the signed
commit. Required CI, CodeRabbit, all nine Advisor reviews and
exact-commit image publication completed. Twenty-two of 23 recommended
E2E selectors passed. The protected-runtime selector failed during NIM
model download because the upstream runner exhausted disk space; Rebecca
accepted that specific upstream failure as nonblocking. Current-run
cleanup is verified; NIM and subsequent rollback coverage remain
incomplete. Cycle 5 evaluation is complete. Rebecca subsequently
authorized cycle 6 and its mandatory Pi receipt-only follow-up; PR merge
remains unauthorized.

### Cycle 4

- Candidate `92e4cc4cda6cbe72f712df4e62dbb080d96ce1de` repairs six
causes observed on cycle 3: custom HTTPS profile TLS readback, an E2E
fixture import requiring an unbuilt CLI catalogue, native receipt
retention during rebuild, native resume credential recovery, verified
credential migration accounting, and the native Anthropic baseline
input.
- Reuse still requires matching endpoint/API/credential key/gateway
authority. Missing authority and credentials fail before mutation.
Credential retirement applies only to the exact newly persisted value;
reuse and Bedrock adapter tokens do not claim migration. Legacy routes
retain their existing behavior.
- Focused local evidence: profile/probe/fixture 135 tests passed;
registry 62 passed; final recovery/migration sibling 117 passed;
baseline 4 passed; final hook-repair shim/migration/fixture 35 passed.
CLI type checking passed with a 12 GiB heap. Working-tree mock/live
parity reported no errors. Normal commit and publication hooks passed;
the signed commit is GitHub Verified.
- Complete cycle 3 failed-job log review and affected assertion/caller
audit are retained. The new channel ledger covers 563 predicates;
corrected native Anthropic ledger covers 114. No new target, retry,
deadline, assertion-budget or product-scope expansion. Commit-hook
findings were repaired without raising architecture/growth limits.
- Cycle3 [standard E2E
38690740007](https://github.com/NVIDIA/NemoClaw/actions/runs/38690740007)
failed with the grouped causes above; protected runtime was skipped. Its
separate [Launchable
38690773573](https://github.com/NVIDIA/NemoClaw/actions/runs/38690773573)
passed with exact source/image evidence and confirmed workspace absence.
These are historical results, not cycle 4 passes.
- Cycle 4 CI and automated reviews completed. CI found the TLS mock and
stale Pi receipts repaired in cycle 5. All nine Advisor artifacts were
reviewed; public documentation remains with the authorized post-merge
owner, and additional Hermes combination coverage was declined with its
gap recorded. Image publication passed. Runtime results are summarized
under cycle 5 above.

### Cycle 3

- Candidate `77b4c216412ddc16f2b7c559202cadcaa9eb9202` repairs native
onboarding/rebuild smoke and Hermes managed-policy credential handling.
Native checks verify the recorded attachment, endpoint, API, model and
scoped credential reference before inference. Legacy paths remain
explicit.
- Local evidence: Hermes policy reader 74 tests passed; existing smoke
94 passed; native onboard/rebuild shell probes 2 passed.
Route-preservation regressions passed after integrating main #13051. E2E
support request/config/planner tests passed, including the final
selector/planner/growth batch of 147 tests.
- Cloud and Hermes runtime probes now use the recorded route. The
existing OpenClaw Anthropic selector verifies native credential
injection and a real agent request. Local support tests own
deterministic request/config behavior. No additional E2E target, API
matrix, retry allowance, or assertion-budget increase.
- `npm run e2e:assertions:check`: passed, 1,257 direct expectations
across 77 live files. The exact previously approved growth delta remains
unchanged. Normal commit and publication hooks passed, including
ShellCheck, repository checks, growth checks and final CLI/plugin type
checks. GitHub verified the signed commit. Current-head CI/reviews/E2Es
remain required before readiness.
- Completed failed-job log and predicate audit for cycle 2. Its standard
E2E run had ten runtime failures, with production smoke/policy defects
and route-oracle mismatches grouped here. AMD64 encountered an external
archive HTTP 429; the earlier OpenClaw resume Error transition lacks
enough retained diagnostics to attribute. No speculative code or retry
was added for those conditions.
- The existing main conflict was resolved with one merge of
`80491e92e2cda863d6bf42869a8f8a5b0a975628`. Source-route reservation and
backup safety changes from #13051 are preserved.
- This candidate requires fresh CI, CodeRabbit, all nine Advisor
reviews, exact-head images and every applicable recommended E2E. Earlier
results do not establish cycle 3 runtime success.

### Cycle 2

- Candidate `e14836f5d134d146099079773bf52ba0c6fba188` repairs
selected-provider receipt filtering, recovery verification/messages, and
receipt-aware export. Existing legacy fixtures remain explicit.
- Focused CLI regression batch: 558 tests passed across 7 files. After
final export/profile refactoring: 310 tests passed across 3 files.
Reconciliation/growth checks: 32 tests passed. Selected-agent E2E probe
support: 41 tests passed.
- CLI type checking passed with an 8 GiB heap. Normal commit and
publication hooks passed, including the final CLI type check. Cycle 2
required CI and CodeRabbit completed; all nine Advisor artifacts were
collected. The native Anthropic evidence gap is addressed in cycle 3.
Its standard E2E run failed; its separate staging Launchable run passed
and confirmed workspace deletion.
- Genuine AMD64/ARM64 Pi qualification receipts came from [successful
image run
38643585682](https://github.com/NVIDIA/NemoClaw/actions/runs/38643585682);
strict source parity passed. These receipts do not replace current-head
runtime results.
- The added selected-agent route probe checks endpoint, API, selected
model, and opaque credential reference. Rebecca approved its exact
assertion-growth delta in [the maintainer
record](#13201 (comment)).
Independent CI must verify that record. No new E2E target or API matrix
was added.
- Cycle 1 CI exposed the repair batch above. Its manual E2E stopped at
an image-publication prerequisite before runtime tests. All applicable
cycle 2 Advisor-recommended selectors were dispatched. New-head evidence
remains required.

### Cycle 1 local evidence

- Focused endpoint, native lifecycle, inference get, switch, probe,
onboarding, and E2E-support tests: 11 files, 456 tests passed. This
includes the final dependency consolidation and custom route observation
changes.
- Earlier focused lifecycle, cleanup, registry, health, generation, and
E2E-support batch: 14 files, 379 tests passed. The final switch rollback
regression reproduced before repair and passed afterward with all 45
native switching tests.
- `npm run test:projects:check`: passed. `npm run e2e:assertions:check`:
passed after the final fixture consolidation, with 1,256 direct
expectations across 77 live test files. The final E2E probe/support file
passed all 37 tests.
- Normal pre-push hooks passed publication validation and the CLI,
plugin, and checked-JavaScript type checks for commit
`05f5dafe655482077b42bd36c0230c88dcc6ec81`, with
`NODE_OPTIONS=--max-old-space-size=8192`.
- Source architecture budget: passed without changing limits. The normal
commit hooks passed lint, formatting, repository checks, growth checks,
secret scanning, and commitlint.
- Live runtime verification and broad CI are pending. The local
readiness doctor reported the default Node heap limit and insufficient
local Docker allocation; runtime evidence will use the repository GitHub
E2E workflow.
- The diff contains no credentials, API keys, or other secrets. Test
values are synthetic.

## Review notes

### Accepted cycle 5 infrastructure exception

Rebecca explicitly accepted the known upstream disk-exhaustion failure
in protected runtime job
[116419249017](https://github.com/NVIDIA/NemoClaw/actions/runs/38729390261/job/116419249017)
as nonblocking for `9029658852ce5600bbfcfae943be13e265a482cf`. The
failure occurred before NIM agent qualification and subsequent rollback
checks; those checks remain incomplete. The current-run owned-container,
fixture, registry and Docker-auth cleanup evidence passed. Historical
cycle 4 cleanup remains unverified. All active required checks pass;
Advisor findings retain their [recorded
dispositions](#13201 (comment)).

Sensitive paths include the changed files under `agents/**`,
`src/lib/inference/**`, `src/lib/onboard/**`, `src/lib/onboard.ts`, and
`tools/e2e/**`, classified against the canonical sensitive-path policy.
Self-review of NVIDIA/NemoClaw commit
`05f5dafe655482077b42bd36c0230c88dcc6ec81` covered the candidate diff,
ownership and endpoint boundaries, rollback, failure states, and focused
negative tests. No independent pre-publication review or waiver is
claimed. These paths await independent automated review and runtime
evidence.

Cycle 5 self-review covered NVIDIA/NemoClaw
`9029658852ce5600bbfcfae943be13e265a482cf`, including sensitive
`scripts/nemoclaw-start.sh` and `src/lib/sandbox/**` changes: credential
custody, scoped handle equality, endpoint guards, atomic config
mutation, startup handoff, affected callers and negative tests.
CodeRabbit and all nine Advisor specialist reviews are complete, with
source-backed dispositions recorded. Current-commit E2E evidence and the
accepted upstream storage-capacity exception are summarized above.

Cycle 4 self-review covered NVIDIA/NemoClaw
`92e4cc4cda6cbe72f712df4e62dbb080d96ce1de`: the six repair groups, exact
binding and credential-retirement guards, all affected callers, complete
failed logs, and unchanged live assertions/cleanup. Current-head
independent review and runtime evidence remain pending.

Cycle 3 self-review covered NVIDIA/NemoClaw
`77b4c216412ddc16f2b7c559202cadcaa9eb9202`: native attachment and
registry authority, selected config/API/credential checks, Hermes
managed policy, the trusted selector boundary, negative local tests, and
retained cleanup/retry contracts. Independent reviews and live results
for this commit remain pending.

Rebecca subsequently authorized the local merge repair, its publication
and GPU qualification on the next commit. That monitoring cycle
completed and paused. Rebecca then authorized the conflict resolution
and resume repair above, followed by the current selector repair and up
to three further justified revision cycles after complete candidate
evaluation. The protected timeout fix is deferred outside this PR as an
infrastructure fast-follow; no all-pass result or broad waiver is
claimed. PR merge remains unauthorized. Current repair self-review
covers the provider-key mapping, fixed-provider endpoint guards,
custom-native scoped-handle refresh, atomic persistence, and affected
negative tests; independent review for
`94c8ec0b0dd0ada2161719bb082cab0bb693f739` completed with the routing
findings repaired in `19f521416a5dbb62ac9fa169e5f03eb817c72297`. Current
repair self-review checked API defaults, explicit Responses,
namespace/endpoint/credential agreement and shared-route controls.
Independent review of `19f5214` completed with both routing findings
resolved and retained Advisor dispositions. Receipt follow-up
self-review authenticates source, image digests, paired run/attempt,
exact contract bytes, authority hashes and strict source parity.
Independent review of the receipt-only commit completed, including the
single resume defect repaired above.

Resume repair self-review covers NVIDIA/NemoClaw
`53bc89879a90e533a191f8b8df2022dcad092ab6`, specifically
`src/lib/onboard/machine/resume-provider-shim.ts`,
`src/lib/inference/native-provider/custom-setup.ts` and their existing
test owners. It checked credential-name precedence, receipt validation,
provider/endpoint binding, API and live resource identity checks,
no-rotation reuse, alternate legacy/reservation paths and mismatch
failures. Independent automated review of `53bc898` completed with the
source-selection repair and retained policy/gap dispositions above. No
merge approval or new waiver is claimed.

Current merge self-review covers NVIDIA/NemoClaw
`afea1a7c05f3383bd20d5f79a477c0c8c0179423`, the affected
native-local/custom-hosted route owners, credential and endpoint
authority, actual probe request/response boundaries, export/profile
validation, sibling configuration paths, and deterministic source
selection. Independent current-candidate review and live results remain
pending.

---
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Configure custom OpenAI-compatible and Anthropic-compatible endpoints
for native inference, with support for multiple APIs and
provider-specific credentials.
* Custom native routes are supported during onboarding, inference
switching, sandbox rebuilds, and model changes.
* Bedrock-compatible Anthropic endpoints can use the native adapter
route.
* **Bug Fixes**
  * Inference status now reflects the endpoint attached to the sandbox.
* Failed route checks provide guidance when a legacy inference route is
unreachable.
* Failed provider changes restore the previous attachment when possible,
and custom attachments are retained only when their route settings still
match.
* Custom endpoint credentials and routing are checked against the
selected provider configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Comment on lines +91 to +96
await verifyNativeCustomStatusAttachment({
gatewayName: options.gatewayName,
sandboxName: options.sandboxName,
expected: options.attachment,
environment: options.environment,
});
Comment on lines +1148 to +1152
await verifyNativeCustomStatusAttachment({
gatewayName: context.gatewayName,
sandboxName: context.sandboxName,
expected: nativeCustomProviderAttachment,
});
Comment on lines +71 to +72
NativeCustomProviderError,
profilePath,
logicalProvider: NVIDIA_HOSTED_LOGICAL_PROVIDER,
label: "NVIDIA",
},
NativeNvidiaProviderError,

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: providers Inference provider integrations and provider behavior area: security Security controls, permissions, secrets, or hardening feature PR adds or expands user-visible functionality integration: dcode LangChain Deep Code integration behavior integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slice 3: Move user-supplied hosted endpoints to native inference

5 participants