Add Trustabl Agent Scanner to CI - #892
Conversation
|
Thanks for opening your first pull request on RoboCo! Quick checklist before review (most of these are enforced by CI, but worth a glance):
See CONTRIBUTING.md for the full workflow and the Code of Conduct for the community standards we follow. Welcome aboard — a maintainer will review shortly. |
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
There was a problem hiding this comment.
Summary
This PR adds a third-party security scanner (trustabl/trustabl-action) to CI. The workflow is well-structured in places — minimal top-level permissions, job-scoped write grants, SHA-pinned actions, persist-credentials: false — but the core issue is supply-chain trust: it introduces an unvetted third-party action with security-events:write and pull-requests:write into CI, running on every PR including external forks, while the action fetches a mutable-tagged tool artifact (version: v0.1.6) at runtime. The existing workflows use only first-party actions (actions/, github/). Several convention inconsistencies with the repo's established CI patterns (checkout version drift, a concurrency group that doesn't collapse push+PR, no paths filter) should also be addressed before this is mergeable. Requesting changes pending maintainer vetting of the action and alignment with repo conventions.
Findings
| File | Line | Severity | Expected → Actual |
|---|---|---|---|
.github/workflows/trustabl.yml |
30 | major | Third-party GitHub Actions added to CI are vetted by the org and requested by a maintainer; all existing workflows use only first-party actions (actions/, github/). → Introduces trustabl/trustabl-action@973f666d... from an org outside the repo's trust boundary, granted security-events:write and pull-requests:write so it can upload to the Security tab and post PR comments. SHA-pinning prevents tag mutation but does not vet the action's own code. |
.github/workflows/trustabl.yml |
32 | major | Runtime-fetched tool artifacts referenced by CI actions should be pinned to an immutable digest, not a mutable tag. → The action's version: v0.1.6 input is a mutable tag. The SHA-pinned action fetches this version at runtime, so a republished v0.1.6 artifact would execute with the job's write permissions — reintroducing the mutable-tag supply-chain risk inside the pinned action. |
.github/workflows/trustabl.yml |
25 | minor | actions/checkout version consistent with the rest of the repo (every existing workflow pins actions/checkout@v7). → Pins actions/checkout@11bd71901bbe...# v4.2.2, a different version than every other workflow, leaving two checkout versions in CI. |
.github/workflows/trustabl.yml |
13 | minor | Concurrency group that collapses duplicate push+pull_request runs for the same branch, per the repo's established pattern: ${{ github.workflow }}-${{ github.head_ref |
.github/workflows/trustabl.yml |
5 | minor | Triggers scoped with paths: filters so CI doesn't burn runners on irrelevant changes (every existing workflow uses paths:). → Bare pull_request: and push: branches: [master] with no paths: filter runs on every PR and every master push, including docs-only/CHANGELOG-only changes — wasted CI minutes against the org's monthly budget cap. |
.github/workflows/trustabl.yml |
22 | nit | An advisory scanner that is meant to surface findings should produce a visible signal when it runs or breaks. → continue-on-error: true with no required-check or status surface means a broken or empty scanner is invisible — no signal that it's working or producing output. |
Verdict
changes requested
We came across your repo and we like that you're building a unique platform for AI agent collaboration and development. We scanned the repo, and noticed agent runtime reliability findings that might be worth reviewing.
File: roboco/agent_sdk/intake_driver.py, The Claude Agent SDK uses the tool's docstring as the description shown to the model.
Recommendations are based on our understanding of agent runtime reliability, some findings may be intentional. Please let us know if this was intentional or if our findings are helpful so we can improve the accuracy of the scanner.
Best,
Trustabl.ai
Open-source AI agent reliability scanner (runs locally, GitHub Action)