test(uipath-maestro-flow): move the inline-agent flow tasks from uipath-agents and grade outcomes

[tmatup]

Short version: the 16 inline-agent-in-a-flow tasks move from the uipath-agents suite to uipath-maestro-flow. Their pass/fail now depends on what the agent produced, not on which authoring route it took. Since the builder-SDK Flow skill was promoted (#3624, #3672), these tasks have shown uipath-agents red for a skill that team does not own. They failed on v1 command gates the SDK route never runs.

Why

Stefan Andrei Balas raised this in the nightly-results thread for 2026-10-05_04-24-14: "Should these tests be moved under uipath-maestro-flow's evals suite and updated with the SDK in mind?" Dustin Metzgar agreed.

• Every prompt is a Flow authoring task ("Create a UiPath Flow project …"), and the first tool call in every run is Skill uipath:uipath-maestro-flow.
• inline-flow group on the claude nightly: 15/15 (09-30) → 13/15 (10-01) → 9/15 (10-02) → 5/15 (10-05).
• All 10 failures on 10-05 hard-gated on uip agent init/refresh/validate --inline-in-flow and uip maestro flow registry get. The SDK route compiles .flow-sdk/<Name>.flow.ts, which writes agent.json + resources/ itself, so those commands never run.

What changed

• Layout. tests/tasks/uipath-agents/lowcode/inline_* → tests/tasks/uipath-maestro-flow/inline_agent/<same dir>/. The inline-agent checkers and inline_wiring.py move from uipath-agents/_shared/ to uipath-maestro-flow/inline_agent/_shared/ (renamed, imports unchanged). reference.directory: ../.. exposes both these checkers and the suite's _shared/validate_flow.py. A task can mount only one reference dir, so they could not stay in the agents tree.
• Ownership. CODEOWNERS lists the agents team (@UiPath/team-coded-agents + the uipath-agents owners) on inline_agent/ together with @UiPath/maestro-cli-team. The resource.json contract these checkers assert is the agents skill's.
• Tags. uipath-agents → uipath-maestro-flow. lifecycle:generate and resource added where missing (required by tests/README). inline-flow and low-code are kept. task_ids are unchanged so evalboard history carries over (the skill-agent- prefix stays).
• Gates removed: every command_executed on uip agent (init|refresh|validate) … --inline-in-flow and uip maestro flow registry (search|get|list) (3–6 per task).
• Gates made advisory (pass_threshold: 0, required by test_same_ground_corpus.py): uip solution resources list/get/refresh. uip is (is-connector) is route telemetry, so it is now weight: 0.
• Gate added: uip maestro flow validate on the named flow file (weight 3.0). validate_flow.py now takes optional explicit paths, because the brownfield fixtures (e.g. OnboardingFlowSol with EmployeeOnboarding) carry a second Flow project that discovery correctly refuses as ambiguous. Two unit tests were added.
• Kept: the .uipx json checks, file_exists, the brownfield command_not_executed guards, the memory-space hand-write guards and uip agent memory add gate (memory is CLI-only: the SDK refuses memory), and every contract checker.
• Prompts use the suite's canonical headless preamble (test_headless_preamble.py). "Do NOT publish, upload, or deploy." is kept.

Checker semantics

Every contract checker grades what the agent produced. Where the SDK shape differs from the v1 shape, the checker accepts the SDK shape only if a live run showed it works.

• Inline agent definition (all 16 tasks). The inline agent's agent.json must be a usable definition: valid JSON, type: "lowCode", id equal to its directory name, a non-empty settings.model, and non-empty system and user messages (inline_wiring.assert_inline_agent_definition). This replaces the removed uip agent validate --inline-in-flow gate as an interim check. That command rejects every SDK-compiled agent today (agent.json version: 1.1.0 vs latest 1.2.0, and no flow-project bindings_v2.json for tool resources). Tracked in UiPath/flow-builder-sdk#962; once it is fixed, agent validate becomes the gate.
• External tools (check_inline_external_{rpa,apiworkflow,maestro,agent}_tool.py). (location, properties.folderPath) must be one of the two pairs that ran live: ("solution", "") (SDK) or ("external", "<literal folder>") (v1). Mixed pairs fail. The folder must also be in the flow's top-level bindings[] b<Name>FolderPath entry. Evidence, one live flow debug A/B per kind on flow-builder-sdk#922: process, agent, api + maestro. In each, the tenant tool job ran Successful from AgentService with both shapes, and the control with no resources/ had no tool. Not tested: the deployed path (pack + deploy + run), tracked in UiPath/flow-builder-sdk#964.
• Solution tools (check_inline_solution_{rpa,apiworkflow,maestro,agent}_tool.py). folderPath may be "" or "solution_folder". Both produce byte-identical derived solution files ([Discussion] Cross-folder review of tests/tasks/ prompts — natural-user voice vs. eval-harness leakage #922).
• IS connector (check_inline_is_connector_tool.py). The solution-level resources/solution_folder/connection/*.json is not required, because pack and debug write it themselves. The checker requires a bindings_v2.json connection binding for the node's connector, and its ConnectionId must be a real UUID. A placeholder string or the SDK's all-zero stub fails.
• Everything else is unchanged, including every "resource.json must exist" assertion and the solution escalation's folderName: "solution_folder".
• Unit tests for the new pass/fail logic: inline_agent/_shared/test_inline_wiring.py (16 cases). It runs the real checker scripts on temp-dir fixtures and is collected by pytest tests/tasks/uipath-maestro-flow/.

Escalation app swap (inline_external_escalation)

The task now escalates to ExpenseReviewApp (Workflow Action, Shared/uipath-agents/ExpenseReviewSol) instead of FraudEscalation.

• FraudEscalation is a JS Action app. A JS Action app cannot back an agent escalation: uip solution resources refresh reports it "not found in Orchestrator", and the agents skill says only Workflow Action apps can. With it, the task could only grade files that would never run.
• ExpenseReviewApp backed a live escalation in UiPath/flow-builder-sdk#925: flow debug suspended and an Action Center AppTask was created.
• The scenario moves from fraud triage to expense-claim triage so the prompt fits the app (input Content, in/out Comment, outcomes approve/reject). Project/solution: ExpenseFlow / ExpenseFlowSol. The checker's expected appName / folderName follow; every other assertion is unchanged.
• The checker passes on [Discussion] Prompt review — tests/tasks/uipath-admin/ #925's compiled output (the run that created the AppTask), and the task passes on this branch (below).

Expected state after merge

Branch run 37541450751 (agent claude, head 2eb614285; the only later commit, 6fbea541b, changes a comment): 14 of 15 runnable tasks pass, against 5 of 15 in nightly 2026-10-05_04-24-14. inline_mcp_server_tool is skipped (skip: true).

task	result	score
in-flow, memory-space, builtin-tool, external-escalation	✅	1.000
external-rpa / solution-maestro	✅	0.966
context-index	✅	0.963
solution-agent-tool	✅	0.931
solution-rpa / solution-apiworkflow	✅	0.923
external-agent / external-apiworkflow / external-maestro	✅	0.862
is-connector-tool	✅	0.857
solution-escalation	❌	0.577

Scores below 1.0 on passing tasks come from the advisory (pass_threshold: 0) uip solution resources route criteria. Every gated criterion passed.

• builtin-tool and external-escalation now pass because flow-builder-sdk#920 and docs(uipath-maestro-case): close send_as drop regression via reinforced required-input gate #921 shipped in the eval image.
• solution-escalation fails one assertion: the channel's properties.folderName is "", and the checker expects "solution_folder". This is the same "" vs "solution_folder" question [Discussion] Cross-folder review of tests/tasks/ prompts — natural-user voice vs. eval-harness leakage #922 answered for solution tools. It has not been measured for escalations, so the checker stays strict. Tracked in UiPath/flow-builder-sdk#967.

Nightly selection: the nightly runs the whole tests/tasks/ tree. Its 2026-10-06 run executed 1398 tasks, including all 17 skill-agent-inline-* task_ids, and skipped tasks under 9 different skill folders. So the new uipath-maestro-flow/inline_agent/ path is picked up with no config change. Maturity skipping keys on task_id plus a hash of the task definition (eval_runner/skills/maturity.py), so history carries over, and the edited definitions re-run instead of being skipped as mature.

Checks run

• pytest tests/tasks/uipath-maestro-flow/: 1683 passed. pytest tests/tasks/uipath-agents/: 17 passed.
• scripts/check-task-driver.py, scripts/check-task-host-paths.py: OK.
• coder-eval run of all 16 moved tasks on this branch (agent claude): 37541450751, table above.
• Studio Web flavor: no experiment runs skill-flavors/studioweb, so no eval exercises the classic v1 Flow skill. Dropping the v1 command gates removes no live coverage.

The inline_mcp_server_tool task (skipped, e2e-blocked) carries path-to-ga. Its moved YAML shows the tag as an added line, so the Path-to-GA approval gate needs an allowlisted reviewer (e.g. Rocky Madden).

🤖 Generated with Claude Code

https://claude.ai/code/session_01QzeWmKmvxJeWdkrXhGPgUE

[github-actions]

Claude finished @tmatup's task in 4m 10s —— View job

---

PR Review: test(uipath-maestro-flow) — move inline-agent flow tasks

• Read review criteria and project conventions
• Analyze full diff
• Read key changed files for surrounding context
• Check CODEOWNERS, tags, task structure
• Check checker logic changes
• Post review findings

---

Summary

Moves 16 inline-agent-in-a-flow test tasks from uipath-agents to uipath-maestro-flow, reframes grading from v1 CLI trajectory gates to outcome-based checks (validate + contract checkers), and relaxes the external-tool checker location/folderPath assertions based on a live flow debug A/B experiment. The rationale is sound — these tasks trigger the Flow skill, and the SDK route produces valid artifacts that the old v1 gates rejected.

Change-by-Change Review

1. YAML comment swallows next list entry — 6 occurrences across 5 task files

Severity: Critical

In several greenfield task YAMLs, a trailing # comment on a pass_threshold: line extends to the end of the line and includes what should be the start of the next - type: criterion. YAML treats everything after # as a comment, so the subsequent criterion's keys (description, tool_name, command_pattern, weight, pass_threshold) become additional/overriding keys on the previous criterion's mapping rather than a new list entry.

Affected files and lines:

File	Line	Swallowed criterion
inline_external_rpa_tool.yaml	57	resources get merged into resources list
inline_external_apiworkflow_tool.yaml	58	resources get merged into resources list
inline_external_maestro_tool.yaml	59	resources get merged into resources list
inline_external_agent_tool.yaml	56	resources get merged into resources list
inline_is_connector_tool.yaml	51	solution init merged into uip is
inline_is_connector_tool.yaml	77	file_exists for flow file merged into resources refresh

The last one is the most impactful — the file_exists check for the flow file is silently lost, and the resources refresh criterion ends up with type: file_exists (overwritten by the comment-swallowed entry), path:, description: from the file_exists block, plus leftover command_pattern/tool_name from the original — a malformed hybrid that the coder_eval framework may misinterpret.

Since all affected criteria have pass_threshold: 0.0 (advisory), this doesn't change pass/fail outcomes for the 4 external-tool tasks. But for inline_is_connector_tool.yaml the file_exists criterion (which had pass_threshold: 1.0) is silently eaten — that does weaken the grading.

Fix: Add a newline before each swallowed - type:. Example for inline_external_rpa_tool.yaml:57:

    pass_threshold: 0.0  # advisory: discovery route is not graded; the contract checker grades the resource it produced

  - type: command_executed

Fix this →

2. CODEOWNERS addition

Severity: OK

New entry at line 166 covers /tests/tasks/uipath-maestro-flow/inline_agent/ with both @UiPath/team-coded-agents (agents owners) and @UiPath/maestro-cli-team plus individual reviewers. The comment explains why both teams own this path. Correct.

3. tests/tasks/uipath-agents/_shared/__init__.py — docstring update

Severity: OK

Updated the module docstring to note that inline-flow helpers moved to uipath-maestro-flow/inline_agent/_shared/, and that CODEOWNERS keeps agents team as co-owner. Clean pointer.

4. Cross-file comment path updates

Severity: OK

• tests/tasks/uipath-insights/rbac/smoke.yaml:45 — updated stale path reference from uipath-agents/lowcode/inline_solution_agent_tool to the new location.
• tests/tasks/uipath-maestro-bpmn/_shared/check_solution_select.py:27 — same path update. Both are in comments, both are correct.

5. validate_flow.py — explicit paths mode

Severity: OK

main() now accepts optional explicit flow paths, bypassing discovery. Two unit tests added (test_explicit_paths_skip_discovery, test_explicit_missing_path_fails_without_running). The change is clean — default behavior (no args) is unchanged, and the if __name__ block passes sys.argv[1:]. This is needed because brownfield fixtures with multiple Flow projects make discovery ambiguous.

6. inline_agent/_shared/__init__.py — new package init

Severity: OK

Good docstring explaining the import pattern, the sys.path.insert convention, and the reference.directory: ../.. mount. Clean.

7. inline_wiring.py — shared helpers (moved + extended)

Severity: OK

The moved inline_wiring.py adds:

• INLINE_TOOL_LOCATIONS = ("solution", "external") — both accepted per the A/B evidence.
• INLINE_SOLUTION_FOLDER_PATHS = ("", "solution_folder") — both accepted.
• assert_tool_folder_binding() — validates the flow's top-level bindings[] carries the folder for external tools.

Logic is sound. The evidence link to flow-builder-sdk#922 is consistently cited. No cross-skill imports.

8. External tool checkers (rpa, apiworkflow, maestro) — location/folderPath relaxation

Severity: OK

The 3 external-tool checkers now accept location in INLINE_TOOL_LOCATIONS (both "solution" and "external") and validate the folder binding via assert_tool_folder_binding(). The folderPath may be "" (SDK shape) or the literal folder (v1 shape). This is a well-justified evidence-backed relaxation.

The 3 solution-tool checkers similarly accept folderPath in INLINE_SOLUTION_FOLDER_PATHS ("" or "solution_folder"). Correct.

9. Task YAML files — structural review

Severity: OK (except the Critical finding in §1)

All 16 tasks:

• tags start with uipath-maestro-flow ✓
• task_id preserved with skill-agent- prefix (evalboard history continuity) ✓
• reference.directory: ../.. consistently set (reaches suite _shared/) ✓
• initial_prompt includes the canonical headless preamble ✓
• uip maestro flow validate outcome gate added (weight 3.0) — clean ✓
• Contract checker run_command (weight 5.0) as primary gate — correct ✓
• Advisory criteria (pass_threshold: 0.0) for route-telemetry commands — correct ✓
• inline_mcp_server_tool kept skip: true with e2e-blocked and path-to-ga tags ✓

10. inline_external_agent_tool checker — strict location: "external" kept

Severity: OK

The PR description explains this was intentional — the A/B only covered process tools, not kind: 'agent'. Correct conservative choice.

What's Missing

• The 6-instance comment-swallows-entry YAML bug needs fixing before merge (see §1). For the 4 external-tool tasks the impact is cosmetic (advisory criteria only), but for inline_is_connector_tool it silently drops a gating file_exists criterion.
• Nothing else identified — CODEOWNERS is updated, old paths are cleaned up, cross-references are updated, no secrets or personal paths.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	These are task YAMLs, not skill SKILL.md files
E2E Tests	Issue	6 YAML entries swallowed by trailing comments (Critical §1)
Skill Body	N/A	No skill body changes
References & Assets	OK	Checker scripts and fixtures correctly relocated
Repo Hygiene	OK	CODEOWNERS updated, stale path refs fixed, no secrets

Issues for Manual Review

• Evalboard run validation: The PR description links a coder-eval run (actions/runs/37358911886) whose results should be posted as a comment. Verify that the moved tasks produce results consistent with expectations before merge.
• path-to-ga tag on inline_mcp_server_tool: The PR description notes this needs an allowlisted reviewer approval.
• Brownfield fixture coverage: The inline_solution_maestro_tool fixture (OnboardingFlowSol) includes a second project (EmployeeOnboarding) that is a BPMN project. The new validate_flow.py explicit-paths mode addresses the ambiguity, but verify manually that the checker finds the correct (newly created) flow, not the fixture's.

Conclusion

The migration rationale is well-justified, the outcome-based grading shift is the right direction, and the checker relaxations are backed by live evidence. Request changes for the Critical YAML comment-swallows-entry bug in 6 places across 5 files — this needs the missing newlines before each - type: that currently sits inside a # comment. Once fixed, the PR looks ready to merge.

---

|

[github-actions]

Claude finished @tmatup's task in 3m 55s —— View job

---

Coder-eval task lint (advisory)

17 task YAMLs changed (1 modified, 16 renamed with modifications); verdict counts: 0 Critical, 0 High, 14 Medium, 1 Low, 2 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

✅ Author confirms passing run: "coder-eval run of all 16 moved tasks on this branch (agent claude): https://github.com/UiPath/skills/actions/runs/37358911886" and "Verification against the 10-05 artifacts with the moved checkers: inline_solution_rpa_tool and inline_external_rpa_tool now pass" and "pytest tests/tasks/uipath-maestro-flow/: 1667 passed."

Per-task lint

tests/tasks/uipath-insights/rbac/smoke.yaml — verdict: OK

Comment-only change (cross-reference path updated to the new inline_agent/ location). No rubric issues.

tests/tasks/uipath-maestro-flow/inline_agent/inline_mcp_server_tool/inline_mcp_server_tool.yaml — verdict: OK

Skipped task (skip: true, tier e2e-blocked). Validate-only axis does not apply to non-e2e tiers.

tests/tasks/uipath-maestro-flow/inline_agent/inline_is_connector_tool/inline_is_connector_tool.yaml — verdict: Low (theme-captured; see Theme 1)

Description explicitly documents rationale: "Verifies design-time artifacts only (no runtime connector invocation)" → carve-out downgrades High to Medium, then theme capture downgrades to Low.

The following 14 tasks — verdict: Medium (theme-captured; see Theme 1)

Task
inline_agent/inline_builtin_tool/inline_builtin_tool.yaml
inline_agent/inline_context_index/inline_context_index.yaml
inline_agent/inline_external_agent_tool/inline_external_agent_tool.yaml
inline_agent/inline_external_apiworkflow_tool/inline_external_apiworkflow_tool.yaml
inline_agent/inline_external_escalation/inline_external_escalation.yaml
inline_agent/inline_external_maestro_tool/inline_external_maestro_tool.yaml
inline_agent/inline_external_rpa_tool/inline_external_rpa_tool.yaml
inline_agent/inline_in_flow/inline_in_flow.yaml
inline_agent/inline_memory_space/inline_memory_space.yaml
inline_agent/inline_solution_agent_tool/inline_solution_agent_tool.yaml
inline_agent/inline_solution_apiworkflow_tool/inline_solution_apiworkflow_tool.yaml
inline_agent/inline_solution_escalation/inline_solution_escalation.yaml
inline_agent/inline_solution_maestro_tool/inline_solution_maestro_tool.yaml
inline_agent/inline_solution_rpa_tool/inline_solution_rpa_tool.yaml

All paths relative to tests/tasks/uipath-maestro-flow/.

Within-PR duplicates

No duplicate clusters detected. All 16 renamed tasks exercise materially distinct inline-agent resource types (builtin tool, context index, RPA/API/Maestro/agent tools in external vs solution locations, escalation, IS connector, MCP server, memory space). The shared YAML scaffold is good template reuse per rubric — each has its own contract checker targeting a different resource.json shape and flow-node type.

Themes

Theme 1 — Validate-only flow tests miss correctness (High). 15 of 16 inline-agent tasks are e2e tier with no command_executed matching flow\s+debug. The validate_flow.py gate (uip maestro flow validate) checks JSON schema validity only — a wrong flow that happens to be schema-valid would pass. The contract checkers (check_inline_*.py) verify resource.json shape and flow-node wiring, which is strong design-time coverage, but no criterion exercises the flow at runtime.

This is a known, deliberate trade-off documented in the PR body: the SDK route's runtime coverage depends on flow-builder-sdk fixes (#920, #921, #922) that haven't landed yet. Adding flow debug criteria now would make all 15 tasks fail with no actionable fix. Consider adding flow debug criteria (even as advisory pass_threshold: 0) once the SDK gaps close, or adding a rationale line to each task's description field (like inline_is_connector_tool already has) to formalize the carve-out.

Suggested fix (when ready): Add to each task:

  - type: command_executed
    description: "Advisory: agent ran flow debug to verify runtime behavior"
    tool_name: "Bash"
    command_pattern: 'uip\s+maestro\s+flow\s+debug'
    min_count: 1
    weight: 2.0
    pass_threshold: 0.0  # advisory until SDK gaps close

Notes

• CLI verb reachability axis was skipped — python3 scripts/check-cli-verbs.py requires sandbox approval not available in this run.
• Near-duplicate analysis: no issues. Each external/solution pair (e.g. external_rpa_tool vs solution_rpa_tool) exercises a different code path (greenfield vs brownfield), different fixtures, and different checker assertions (location: "external" vs location: "solution", presence of brownfield command_not_executed guards).

Conclusion

⚠ 15 task(s) have issues, max severity High (theme-level). All per-task verdicts are Medium or lower after theme-aware downgrade. The single theme (validate-only e2e) is a known, documented product-gap trade-off — not actionable until flow-builder-sdk fixes land. Advisory only — not blocking merge.

---

[tmatup]

Review triage (Claude Code Review + task lint):

• §1, Critical: YAML comment swallowed the next criterion (6 places). Confirmed and fixed in f8d58f3. The advisory rewrite's \s*$ consumed the newline before the next - type:. I diffed every moved task against its origin/main original. Each now has old − intended removals + validate gate criteria, and none was lost. The is-connector flow file_exists gate is back.
• Manual review, eval run. Run 37358911886 started on 80d5344, before the fix. For the four external tasks the merged criteria were advisory only. For inline_is_connector_tool the run is missing the file_exists gate. Results will be posted here when the run completes.
• Manual review, path-to-ga. This is expected. The tag on inline_mcp_server_tool (skipped, e2e-blocked) is unchanged, but the move shows it as an added line. It needs an allowlisted approver.
• Manual review, brownfield inline_solution_maestro_tool. The validate gate names the new flow explicitly (OnboardingFlowSol/OnboardingFlow/OnboardingFlow.flow), so the fixture's EmployeeOnboarding project is never selected. The contract checker reads the same fixed path.
• Lint Theme 1, validate-only e2e (no flow debug). Not adding an advisory command_executed on flow debug. It grades whether the command ran, not what it produced, and the suite contract keeps route telemetry weightless. The right runtime gate is a checker that runs flow debug and asserts the tool or escalation was invoked. That belongs after flow-builder-sdk#920/docs(uipath-maestro-case): close send_as drop regression via reinforced required-input gate #921 land, because builtin tools and escalations are inert at runtime today. [Discussion] Cross-folder review of tests/tasks/ prompts — natural-user voice vs. eval-harness leakage #922's live A/B showed what that check needs to witness: a tool job created by AgentService.

[tmatup]

coder-eval results on this branch (agent claude): 10 of 15 runnable tasks pass, against 5 of 15 in nightly 2026-10-05_04-24-14. inline_mcp_server_tool is skipped (skip: true).

Run 37358911886 (head 80d5344) and run 37362751055 (head f8d58f3, re-ran the two tasks the comment bug touched):

task	result	why it fails
in-flow, memory-space, context-index	✅
external rpa / apiworkflow / maestro	✅	the #922 checker change
solution rpa / apiworkflow / maestro	✅	the #922 checker change
builtin-tool	❌	no resources/: flow-builder-sdk#920
external-escalation, solution-escalation	❌	no resources/: flow-builder-sdk#921
external-agent-tool	❌	checker requires location: "external"; the SDK emits "solution" for kind: 'agent'
solution-agent-tool	❌	checker requires folderPath: "solution_folder"; the SDK emits ""
is-connector-tool	❌	resources/solution_folder/connection missing: the agent did not run uip solution resources refresh

The two agent-tool rows are the same shape question as #922. #922's live A/B covered kind: 'process' only, so I left those checkers strict. A matching A/B for an agent tool is needed before relaxing them. On 80d5344 the is-connector task was rejected at load (the comment bug), so f8d58f3 is its first real run. It needs its own look: either the SDK route should provision the solution connection resource, or the skill should tell the agent to run uip solution resources refresh.

[tmatup]

@andreibalas-uipath: a5fcc80 swaps inline_external_escalation's app from FraudEscalation (JS Action) to ExpenseReviewApp (Workflow Action, Shared/uipath-agents/ExpenseReviewSol). A JS Action app can't back an agent escalation, so with FraudEscalation the task could only ever grade files. ExpenseReviewApp created a real Action Center task in UiPath/flow-builder-sdk#925. The scenario changes to expense-claim triage to match the app. Is this swap fine for the agents team? Note that the standalone uipath-agents/lowcode/external_escalation task still uses FraudEscalation; that one is outside this PR.

[tmatup]

CI status: both red required checks come from outside this PR's diff.

• Run skill smoke tests: the smoke image installs @uipath/cli@dev (1.204.x), but maestro-tool, solution-tool and automation-hub-tool at @dev are already on 1.205.0-dev. The CLI refuses tools from another line (Tool 'maestro' runs @uipath/maestro-tool@1.205.0-dev.9021, which is off this CLI's 1.204.x line). Image setup then reports registry pull failed / source check failed, and Flow and BPMN smoke tasks fail across PRs; none of the failing tasks are the moved inline_agent tasks. A re-run failed the same way. Fix: fix(tests): pin the smoke image's tool plugins to the CLI's release line #3762 (pins the image's tools to the CLI's release line). I'll merge main in and re-run smoke once it lands.
• path-to-ga approval: inline_mcp_server_tool already carried path-to-ga; moving the file makes the gate see it as an addition. It needs an approval on the current head from an allowlisted reviewer (@rockymadden or another on the list).

Code-owner approvals still needed under the branch rules: @UiPath/maestro-cli-team (tests/tasks/uipath-maestro-flow/, tests/tasks/uipath-maestro-bpmn/_shared/) and @UiPath/insights (a one-line path fix in a comment in tests/tasks/uipath-insights/rbac/smoke.yaml).

🤖 Generated with Claude Code
Co-Authored-By: Claude

[rockymadden]

Recommendation: 🟡 Commented. The move is complete and nothing was dropped. The relaxed checkers rest on live A/B evidence for 2 of 4 tool kinds. Removing uip agent validate left no check on agent.json. The PR description no longer matches the head.

What

• 📦 Moves the 16 uipath-agents/lowcode/inline_* tasks, their 15 _shared checkers, inline_wiring.py and inline_in_flow/check_inline_in_flow.py to uipath-maestro-flow/inline_agent/. task_ids are unchanged.
• ✂️ Removes the v1 trajectory gates (uip agent init|refresh|validate --inline-in-flow, uip maestro flow registry). Adds a uip maestro flow validate gate (weight 3.0) on each named .flow.
• 🔓 External tool checkers now accept location solution|external and folderPath ""|literal, and look for the folder in the flow's top-level bindings[]. Solution tool checkers accept folderPath ""|solution_folder.
• 🔁 The is-connector checker drops the solution connection/ resource. It now requires a matching bindings_v2.json connection binding.
• 🔁 The external escalation task swaps FraudEscalation (JS Action app) for ExpenseReviewApp (Workflow Action app).
• Why: the Flow skill now authors through the SDK, which never runs the v1 commands. That made uipath-agents red for a skill that team doesn't own.

Overall findings

• 🟠 major: the PR description no longer matches the head.

  • It says the agent-tool checkers "stay strict until an agent-tool A/B settles it", and lists inline_external_agent_tool / inline_solution_agent_tool as expected failures. 1d2f62923 relaxed both.
  • It lists inline_is_connector_tool as failing on the missing resources/solution_folder/connection. aabb7beff removed that assertion.
  • "Expected state after merge" and "10 of 15" were measured before those commits. They are claims about older code.
  • Fix: rewrite "Checker semantics" and "Expected state". Re-run the 16 tasks on the current head and post the new table.

• 🟠 major: nothing grades agent.json any more.

  • The removed uip agent validate --inline-in-flow gate was the only signal on the inline agent's own definition. None of the moved checkers read agent.json.
  • Example: check_inline_in_flow.py passes when inputs.source points at an empty UUID dir, as long as flow validate passes.
  • I could not verify that uip maestro flow validate checks the inline agent's directory.
  • Fix: have the grader run uip agent validate <dir> --inline-in-flow as a run_command outcome gate. Or reuse the suite's _shared/check_inline_agent.py --check exists "<Sol>/<Flow>/*/agent.json".

• 🟡 minor: the new checker logic has no unit tests.

  • assert_tool_folder_binding (inline_wiring.py:209) and assert_connection_binding (check_inline_is_connector_tool.py:149) are new pass/fail logic. "Checked locally" is not in CI.
  • pytest tests/tasks/uipath-maestro-flow/ collects nothing for inline_agent/_shared/.
  • Fix: add inline_agent/_shared/test_inline_wiring.py with fixtures for each case: SDK shape passes, v1 shape passes, wrong binding folder fails, missing binding fails, placeholder ConnectionId fails.

• 🟡 minor: repo comments tell the history of the change rather than what a future reader needs.

  • The same 4-line "Outcome gate ... replaces the ... gates the v1 recipe required" block is pasted into all 16 YAMLs (e.g. inline_in_flow.yaml:72).
  • Other places: the "until 2026-10-05" history in check_inline_external_escalation.py:31-36, the "moved from" note in CODEOWNERS:167-169, and uipath-agents/_shared/__init__.py:23-26.
  • The #922 rationale appears in inline_wiring.py and again in 6 checker docstrings.
  • Fix: keep the evidence link once, on the inline_wiring.py constants. Drop the move/history prose.

• 🟡 minor: could not verify nightly selection.

  • The tag moved from uipath-agents to uipath-maestro-flow. The nightly VM's daily.sh is not in the repo, so I can't confirm it globs tasks/uipath-maestro-flow/inline_agent/**, or that evalboard keys history on task_id and not on path or tag.
  • Fix: name the nightly glob or config that picks these up.

tl;dr: The move is clean and every file landed. Before approval: fix the stale description, put an agent.json check back as an outcome gate, and unit-test the two new assertion helpers. Inline notes cover extrapolated relaxations, a placeholder-accepting connection check, and two stale comments.

[tmatup]

Review Response (@rockymadden, review 5434740761)

Short version: all five body findings and the five inline threads are addressed in 80cb16f and 6fbea54. The branch now passes 14 of 15 runnable tasks on the current head. The one red task is a new, unmeasured shape question, filed as flow-builder-sdk#967. Instead of re-adding uip agent validate, I added a structural agent.json check, because agent validate rejects every SDK-compiled agent today (measured below; flow-builder-sdk#962).

1. The PR description no longer matches the head

Fixed. I rewrote "Checker semantics" and "Expected state" against the current code. Fresh 16-task run on head 2eb614285: 37541450751, agent claude. 14/15 pass (5/15 in nightly 10-05). builtin-tool and external-escalation are green now that flow-builder-sdk#920/#921 shipped. The only failure is solution-escalation: the compiled channel has folderName: "" and the checker expects "solution_folder". That is the same question #922 answered for solution tools, but it has not been measured for escalations, so the checker stays strict (flow-builder-sdk#967). The per-task table is in the description.

2. Nothing grades agent.json any more

Fixed with an interim check; the real gate is blocked on flow-builder-sdk#962. I measured your first option before adding it. I ran uip agent validate <dir> --inline-in-flow (CLI + agent-tool 1.205.0-dev.9071) on the 14 SDK-compiled inline agents from the earlier branch run:

• 13/14 fail AgentValidationOutdated: Agent version "1.1.0" is outdated; latest is "1.2.0". The SDK hard-codes 1.1.0. The one pass is memory-space, where uip agent memory add rewrote the file.
• With version bumped in a copy, 9/14 still fail AgentValidationDrift (<FlowProject>/bindings_v2.json: missing). Those 9 are every agent with a process, api, maestro, agent or context tool. The SDK writes that sidecar only for connections.

As a gate, it would have turned 13 of 14 correct-by-runtime outputs red on an SDK and agent-tool contract gap. So the fix belongs in the SDK (or in agent validate), filed as flow-builder-sdk#962. Until then, inline_wiring.assert_inline_agent_definition runs in all 16 checkers through resolve_inline_agent_dir (plus check_inline_in_flow.py and the is-connector checker). It requires valid JSON, type: "lowCode", an id equal to the directory name, a non-empty settings.model, and non-empty system and user messages. Since flow-builder-sdk#931 the prompts live only in agent.json, so an empty UUID dir or a prompt-less file now fails. The docstring marks the check as interim and links #962.

3. The new checker logic has no unit tests

Fixed. I added inline_agent/_shared/test_inline_wiring.py (16 cases). Each case builds a temp-dir solution and runs the real checker script with it as cwd, the way coder-eval does. Cases: SDK shape passes, v1 shape passes, the three unwitnessed (location, folderPath) pairs fail, a wrong binding folder fails, a missing binding fails, a missing or system-prompt-less agent.json fails, a real ConnectionId passes, and a placeholder, TODO, the SDK stub, empty and null all fail. pytest tests/tasks/uipath-maestro-flow/ collects them (1683 passed).

4. Repo comments tell the history of the change

Fixed. The 4-line "Outcome gate … replaces the … v1 recipe" block in all 16 YAMLs is now one line: # Outcome gate: the authored flow validates, whichever route wrote it. I dropped the "until 2026-10-05" story in check_inline_external_escalation.py (the rule stays: a JS Action app cannot back an escalation), the "moved from" note in CODEOWNERS, and the move note in uipath-agents/_shared/__init__.py. The #922 rationale now appears once, on the inline_wiring.py constants, and the 8 checker docstrings point at it.

5. Could not verify nightly selection

Verified, no change needed. The nightly runs the whole tests/tasks/ tree. Run 2026-10-06_04-22-40 executed 1398 tasks, including all 17 skill-agent-inline-* task_ids, and its skip list spans 9 skill folders. So the new uipath-maestro-flow/inline_agent/ path is in scope with no config change. Maturity skipping (coder_eval_uipath eval_runner/skills/maturity.py) keys on task_id plus a hash of the task definition. History carries over, and the edited definitions re-run instead of being skipped as mature.

Inline threads (each has a reply with details):

• inline_wiring.py:205, api/maestro evidence: Fixed. I ran the flow debug A/B for WeatherAPI and ProcurementProcess (#922 comment). The tool job ran Successful from AgentService in both arms, and the control had no tool. The deployed-path A/B is tracked in flow-builder-sdk#964.
• check_inline_external_rpa_tool.py:118, independent checks: Fixed. Only the witnessed pairs are accepted (assert_external_tool_shape), in all four external checkers.
• check_inline_is_connector_tool.py:169, placeholder ConnectionId: Fixed. A UUID is required, and the all-zero SDK stub is rejected too.
• inline_external_escalation.yaml:64: Fixed, comment deleted.
• inline_is_connector_tool.yaml:6: Fixed, "uip is" clause dropped.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

[tmatup]

CI update (head 6fbea54):

• Run skill smoke tests: green on re-run. fix(tests): pin the smoke image's tool plugins to the CLI's release line #3762 (tool plugins pinned to the CLI's line) merged at 20:58 on 10-06, so the earlier cause is gone. Attempt 1 of run 37541511740 scored 93.8% (61/65) against the 95% gate. The 4 failures were in tasks this PR does not touch, and each was an agent-run outcome, not a setup error: skill-agents-solution-verb-probe (600 s turn timeout), skill-agent-coded-conversational-langgraph and -llamaindex (40-turn cap), and skill-flow-voice-inbound-call (llm_judge 0.0). Attempt 2 on the same commit passed at 98.5% (64/65). The 11 Data Fabric tasks are left out of the rate in both attempts because the tenant's Data Service connection preflight fails.
• path-to-ga approval: still red, by design. The check needs an approval on the current head commit from one of sherifmak, uisherif, bai-uipath, uipreliga, rockymadden. It flags inline_mcp_server_tool.yaml, whose tags: line gains path-to-ga in the move. The existing approvals (andreibalas-uipath, dmetzgar) are not on that list.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xi6SSr1opmH8Wn9F5mexxW