Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
365 changes: 365 additions & 0 deletions apps/electron/e2e/evidence-passage-live-acceptance.ts

Large diffs are not rendered by default.

857 changes: 857 additions & 0 deletions docs/acceptance/evidence-passage/final-report.json

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions docs/acceptance/evidence-passage/finished.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"id": "research-064770ff-2e25-42dd-bc18-e93d72d4ff3e",
"symbol": "NVDA.US",
"startedAt": 1790605880470,
"plannedCapabilities": [
"company.profile",
"market.quote",
"market.kline",
"company.valuation",
"company.financials",
"company.earnings",
"company.ratings",
"research.news",
"market.capitalFlow",
"portfolio.positions"
],
"completedCapabilities": [
"research.news"
],
"failedCapabilities": [
"company.profile",
"market.quote",
"market.kline",
"company.valuation",
"company.financials",
"company.earnings",
"company.ratings",
"market.capitalFlow",
"portfolio.positions"
],
"status": "partial",
"locale": "en-US",
"recoveryCount": 0,
"reportId": "report-research-064770ff-2e25-42dd-bc18-e93d72d4ff3e",
"finishedAt": 1790605911615,
"recoverable": false
}
16 changes: 16 additions & 0 deletions docs/acceptance/evidence-passage/model-usage.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"model": "deepseek-v4-pro",
"usage": {
"prompt_tokens": 2068,
"completion_tokens": 2635,
"total_tokens": 4703,
"prompt_tokens_details": {
"cached_tokens": 0
},
"completion_tokens_details": {
"reasoning_tokens": 1926
},
"prompt_cache_hit_tokens": 0,
"prompt_cache_miss_tokens": 2068
}
}
492 changes: 492 additions & 0 deletions docs/acceptance/evidence-passage/retrieved-news.json

Large diffs are not rendered by default.

19 changes: 19 additions & 0 deletions docs/acceptance/evidence-passage/verification.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"mode": "live AgentKernel + ResearchService production path; real model over HTTP; production research.news capability",
"model": "deepseek-v4-pro",
"symbol": "NVDA.US",
"retrieval": {
"kind": "live public RSS feed",
"url": "https://rss.nytimes.com/services/xml/rss/nyt/Business.xml"
},
"runId": "research-064770ff-2e25-42dd-bc18-e93d72d4ff3e",
"reportId": "report-research-064770ff-2e25-42dd-bc18-e93d72d4ff3e",
"runStatus": "partial",
"retrievedNewsCount": 49,
"evidenceRefCount": 1,
"passageCount": 49,
"allOffsetsRelocate": true,
"allEvidenceMappedToPassage": true,
"survivesReload": true,
"scope": "Passage-level evidence location only. Claim identity, bidirectional claim↔evidence mapping and verifier output are separate PRs and are not exercised here."
}
115 changes: 115 additions & 0 deletions docs/evidence-passage-test-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Evidence passage location acceptance report

Issue #13 asks that report evidence resolve to the **actual source passage** rather than a
document-level citation, and that a claim → source jump still land on that passage after the report
is reloaded. This report covers the **passage-location slice** of that work (PR #251). Claim
identity, the verifier verdicts and the bidirectional claim↔evidence mapping are separate PRs and
are explicitly out of scope below.

## Live Deep Research E2E — PASS

Date: 2026-09-28. Environment: Windows 10.0.26200.9550, Bun 1.4.2, Node 22.22.2.
Commit under test: `bbcb4de` — `feat(research): 记录证据的原始 passage 位置,重载后仍可定位`.

Actual command, from the repository root, after loading the model credential into the process
environment:

```sh
FINAGENT_EVIDENCE_NEWS_FEED='https://rss.nytimes.com/services/xml/rss/nyt/Business.xml' \
FINAGENT_EVIDENCE_MODEL='deepseek-v4-pro' \
FINAGENT_EVIDENCE_SYMBOL='NVDA.US' \
DEEPSEEK_API_KEY=… \
bun apps/electron/e2e/evidence-passage-live-acceptance.ts
```

No fixture sits anywhere on this path — the acceptance fails closed without a model credential and
without a reachable retrieval source:

| Stage | Real component that ran |
| --- | --- |
| Retrieval | The **production** `research.news` capability (`createResearchNewsCapability`), with only its `getNews` fetcher injected to read a live HTTPS feed |
| Model | `deepseek-v4-pro` over the real streaming chat-completions API |
| Synthesis | The production `createAgentSynthesizer` driving the real `AgentKernel` |
| Assembly | The production `ResearchService` → `ResearchRunner` → `assembleReport` |
| Persistence | The real `ResearchReportRepository` over `JsonFileStore` |

| Evidence | Observed result |
| --- | --- |
| Research run | `research-064770ff-2e25-42dd-bc18-e93d72d4ff3e` |
| Published report | `report-research-064770ff-2e25-42dd-bc18-e93d72d4ff3e` |
| Run status | `partial` — news succeeded; the nine capabilities absent from the injected registry are explicitly `unavailable` |
| Retrieved articles | 49 live articles (real HTTP response body retained as `news-feed.xml` during the run) |
| Recorded passages | 49 — one per retrieved article |
| Offsets relocate | every passage re-locates to the **same** position, and `sourceText.slice(start, end)` equals the recorded excerpt |
| evidence → passage | every `EvidenceRef` resolves to at least one passage |
| Survives reload | a **fresh** repository instance reads byte-identical passages and every one still relocates |
| Model usage | 2068 prompt + 2635 completion tokens (1926 of them reasoning) |

A representative recorded passage (offsets are 0-based character offsets into
`"<title>\n\n<summary>"`, hence `line: 3, column: 1`):

```json
{
"id": "passage:run-research.news-1790605880681-1:research.news:feed-0-https://www.nytimes.com/2026/09/28/business/nvidia-stock-buyback.html:69-226",
"runId": "run-research.news-1790605880681-1",
"capabilityId": "research.news",
"documentId": "feed-0-https://www.nytimes.com/2026/09/28/business/nvidia-stock-buyback.html",
"canonicalUrl": "https://www.nytimes.com/2026/09/28/business/nvidia-stock-buyback.html",
"excerpt": "The increase comes four months after the chip giant added $80 billion to its buyback program, bringing the total remaining authorized amount to $235 billion.",
"location": { "start": 69, "end": 226, "line": 3, "column": 1 }
}
```

Committed artifacts:

- [Verification record](acceptance/evidence-passage/verification.json)
- [Final model-generated report](acceptance/evidence-passage/final-report.json)
- [Retrieved news payload](acceptance/evidence-passage/retrieved-news.json)
- [Run summary](acceptance/evidence-passage/finished.json)
- [Provider-reported model usage](acceptance/evidence-passage/model-usage.json)

The report and news payload are test output, not independently verified financial analysis.

## Automated checks

| Executed command | Result |
| --- | --- |
| `bun test --isolate packages/shared/src/research packages/core --timeout 20000` | 185 pass / 0 fail; 540 assertions across 21 files |
| `tsc --noEmit` on `apps/electron/e2e/evidence-passage-live-acceptance.ts` | exit 0 |
| `tsc --noEmit` in the core, shared, i18n, ui and electron workspaces | all exit 0 |

`packages/shared` as a whole still reports 22 environment-only failures (21 Longbridge provider
tests needing an external CLI, 1 Langfuse test needing an external endpoint). An A/B run with the
production code reverted produces the identical 22, so they are unrelated to this change; the PR
body carries that baseline.

## Scope and limits

Proven by this acceptance:

- real network retrieval produces locatable passages;
- every recorded offset resolves back to the same position in the real source text, and the
recorded excerpt is exactly the text at that range;
- the evidence → passage direction is usable;
- all of the above survives persistence and a fresh repository reload.

**Not** proven here, because the code does not exist yet:

- **claim identity** and the bidirectional `claim → evidence_refs` / `evidence_ref → claim_ids`
mapping — PR #67, still open;
- **verifier output** — `claim-verifier.ts` (#90) merged as a standalone library with no callers;
nothing wires it into the report path;
- **report UI bidirectional jump** — unclaimed;
- **Longbridge as the retrieval source** — see below.

Retrieval note: the acceptance environment had no Longbridge CLI and no Longbridge credentials, so
the capability's `getNews` fetcher was injected with a live public RSS feed. The production
Longbridge fetcher remains the capability's default and is exercised end to end by
`apps/electron/e2e/research-recovery-kernel.ts` (PR #80). Because `createResearchNewsCapability`
stamps `provenance.provider` as the literal `'longbridge'`, the checkpoint for this run records
`provider: "longbridge"` even though the fetcher was an RSS feed — a pre-existing quirk of the
manifest, not something this PR changes, but worth knowing before treating that field as
authoritative.

The acceptance harness is opt-in and is not part of any CI gate; it needs a real model credential
and a reachable retrieval source to run.
66 changes: 66 additions & 0 deletions packages/core/src/evidence-passage.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/**
* Passage-level evidence location — where a piece of evidence actually sits
* inside the source it was retrieved from.
*
* Issue #13 asks for evidence that points at a concrete passage/span instead of
* a bare URL or a document-level reference. A `SourcePassage` keeps the minimum
* needed to restore that position later: source/document identity, the verbatim
* excerpt, and location metadata describing where the excerpt was found in the
* retrieved text.
*
* Two deliberate constraints:
*
* - The excerpt is verbatim source text. It is never synthesized, summarized or
* re-worded by Folio, and a verifier's explanation can never become one.
* - The location is metadata about a past retrieval, not a promise about the
* present. Re-deriving it against a freshly retrieved document may legitimately
* fail (the source changed); callers must treat that as "unresolvable", never
* as a reason to guess a nearby offset.
*/

/**
* Where a passage sits inside the source text it was located in.
*
* Offsets are 0-based character offsets into the exact text that was searched,
* so `sourceText.slice(start, end)` is the excerpt itself. `line` / `column` are
* 1-based and are what a reader-facing jump needs in order to scroll a document.
*/
export interface PassageLocation {
/** 0-based offset of the first character of the excerpt. */
start: number;
/** Exclusive end offset. `sourceText.slice(start, end)` equals the excerpt. */
end: number;
/** 1-based line number of `start`. */
line: number;
/** 1-based column of `start` within its line. */
column: number;
}

/**
* One locatable span of a source document, together with the evidence-producing
* run it came from.
*
* A passage is deliberately not a claim: it records "this text exists at this
* position in this source". Which claims rest on it is decided elsewhere, which
* is what keeps one source able to support several claims.
*/
export interface SourcePassage {
/**
* Stable id inside the report. Derived from the run, the capability, the
* source document identity and the span — never from array position — so it
* survives report assembly, persistence, reload and export unchanged.
*/
id: string;
/** `CapabilityRunRecord.id` of the run whose result this source came from. */
runId: string;
/** Capability that retrieved the source, e.g. `research.news`. */
capabilityId: string;
/** Provider document identity within the run, e.g. `NewsItem.id`, when known. */
documentId?: string;
/** Canonical public URL of the source document, when it has one. */
canonicalUrl?: string;
/** Verbatim excerpt taken from the source. Never synthesized. */
excerpt: string;
/** Where `excerpt` sits inside the retrieved source text. */
location: PassageLocation;
}
1 change: 1 addition & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -575,6 +575,7 @@ export interface Skill {
export * from './answer-blocks.ts';
export * from './citations.ts';
export * from './capability.ts';
export * from './evidence-passage.ts';
export * from './research.ts';
export * from './thesis.ts';
export * from './alert-rules.ts';
Expand Down
13 changes: 13 additions & 0 deletions packages/core/src/research.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import type { CapabilityProvenance, CapabilityRunStatus } from './capability.ts';
import type { SourcePassage } from './evidence-passage.ts';
import type { SupportedLocale } from './locale.ts';
import type { StrategyId } from './strategy.ts';

Expand Down Expand Up @@ -88,6 +89,18 @@ export interface ResearchReport {
* failed or were unavailable — the report still stands, gaps are explicit.
*/
runStatus: ResearchRunStatus;
/**
* Locatable source passages behind this report's evidence (issue #13): the
* verbatim excerpt each text-bearing source contributed, plus the position it
* occupies in that source, so a claim → source jump can land on the passage
* instead of the source's front page.
*
* Optional, and deliberately additive to `sections[].evidence` rather than a
* replacement: reports persisted before passage location existed omit it, and
* a source that carries no retrievable text simply contributes nothing. An
* absent entry means "no passage recorded", never "no evidence".
*/
evidencePassages?: SourcePassage[];
}

/** Lightweight progress record for the Research UI. */
Expand Down
Loading
Loading