Skip to content

Analysis Data API: write analysis results back onto samples over HTTP - #105

Closed
XuHuang0 wants to merge 1 commit into
CederGroupHub:mainfrom
XuHuang0:analysis-data-api
Closed

XuHuang0 wants to merge 1 commit into
CederGroupHub:mainfrom
XuHuang0:analysis-data-api

Conversation

@XuHuang0

@XuHuang0 XuHuang0 commented Oct 7, 2026

Copy link
Copy Markdown

Based on origin/main 6aed0ef. Opened at your request. Prompted by D7 and D9 in alab_one's
docs/data-architecture.md and the companion docs/alabos-mongo-write-paths.md.

What it adds

An analysis service runs off the control PC. To work without opening Mongo as a peer (P1) it needs three things
AlabOS does not expose, plus one read-only helper:

route does
POST /api/analysis/results/<sample_id> store metadata.analysis.<kind>, and mirror it to Alab(completed) if the sample is archived there
POST /api/analysis/claim/<sample_id> take or renew a lease on one sample, atomically
POST /api/analysis/claim/<sample_id>/release give it back
GET /api/analysis/pending samples carrying a technique's input data but no result matching the caller's params_hash
GET /api/analysis/data-sources what this deployment declares about where its measurements land

One AnalysisView next to SampleView, with the blueprint as a thin adapter over it: one write implementation
behind both the HTTP and in-process paths, which is D9's shape. Additive, and tested.

Design

AlabOS never looks inside a result. write_result stores an opaque object and reads one field,
params_hash, which it only compares for equality. It is a cache validator, like an ETag, so a change to our
refinement output needs no change here.

Where a lab stores its measurements is config, not code. [data_api.input_sources.<name>] declares a source
as a field path plus an optional BSON type, and AlabOS builds the query; the caller sends a name, never a
filter, because accepting an arbitrary Mongo filter over HTTP is the same surface as handing out the connection
string. In config because the field is a fact about a deployment: alab_one and alab-gpss differ today, and an
alab_one change in flight will move its pattern again. Each of those becomes a config edit instead of a PR.

The claim is a lease, not a lock. One conditional update_one, so of two racers exactly one matches. The
expiry is stored on the document, so a crashed worker's claim frees itself and nothing has to notice the crash.
It is a fixed-width ISO-8601 UTC string rather than a BSON Date because $lte then orders correctly by string
comparison; a Date would never compare against the strings the existing off-box worker already writes.

An optional shared token guards the routes. [data_api] token, or ALABOS_DATA_API_TOKEN, compared in
constant time and never echoed back. With nothing set the check does not exist, so merging this changes no
deployment unless you turn it on. We added it because these are write routes and our analysis host is a shared
machine, so a forwarded port is not ours alone. If you would rather use mTLS, a proxy, per-client tokens, or
treat the lab network as the boundary, say so and we will follow it on our side.

The result write goes through SampleView.update_sample_metadata with a dotted key, so {"analysis.xrd": ...}
becomes a $set of metadata.analysis.xrd. Worth flagging because we checked: every existing caller passes a
flat key and no test covers the dotted form, so the nesting is a consequence of how that method builds its update
rather than a stated contract. We rely on it because the alternative is writing the $set ourselves and
bypassing the one method everything else goes through.

The mirror is asymmetric, deliberately. This API copies a result into Alab(completed) when the sample is
already archived there, because a result written after the experiment finished would otherwise be invisible to
the Data page and lost when the live document is pruned. lab_view.update_sample_metadata during a task does not
mirror, and we did not change it: that is a core method on the lab's hot path, and unifying the two belongs in
its own change rather than in an analysis PR.

One operational note: GET /pending is an unindexed scan of samples, fine at current scale but it wants an
index on metadata.analysis.<kind>.params_hash if the collection grows.

Not in this PR

The SEM client's three paths from section 5 of the architecture doc (link to an existing sample, create a
manual sample, measure without persisting). This is the analysis side only: no sample search, no manual-sample
creation, no do-not-persist gate. If the two are meant to share one contract, the natural shape is for "register
a measurement" and "write an analysis result" to be two endpoint groups under one /api/data. We have not built
that and are not the right people to design the instrument-UI half.

Discovery for anything but XRD. pending needs one configured source per technique, and for SEM the spectra
are files in a shared folder, so that query depends on how the transfer records them. The write path is already
technique-agnostic, so a sem_eds or eis result can be written the moment there is one.

Analysis runs off the control PC and needs three things AlabOS does not
expose: a write of metadata.analysis.<kind> onto an existing sample, a
query for which samples still need a given analysis, and an atomic claim
so two workers never refine the same sample twice.

Implements them as a view next to SampleView with the HTTP routes as a
thin adapter, so there is one write implementation behind both, per the
data-architecture doc's D7 and D9. Where a lab stores its measurements
is read from the config file rather than hardcoded here.
@EmmaBrass

Copy link
Copy Markdown

Fold-in note (unified /api/data)

Thanks for #105 — we are folding it into the unified Data API rather than keeping a permanent public /api/analysis prefix.

What landed / is landing on our unified-data-api branch:

  • AnalysisView kept as the one write implementation (D7/D9).
  • HTTP routes moved to /api/data/analysis/* (pending, claim, release, results, data-sources) with the same bodies/query params and status codes.
  • Optional X-AlabOS-Token unchanged.
  • input_sources for alab_one now point at metadata.diffraction_results.logical_key (measurement refs), not inline .xrdml text or twotheta arrays.
  • Desta also needs GET /api/data/samples/<id> and GET .../artifacts/<kind> (new) so raw patterns are streamed via storage config without a Sauron mount.

Please retarget desta clients from /api/analysis/... → /api/data/analysis/... once this ships.

Sample metadata.source (alabos / manual_sem / ephemeral) exists for identity provenance; analysis pending still keys off measurement input_sources, not source.

— Emma (via Cursor on Sauron)

@EmmaBrass

Copy link
Copy Markdown

Closing without merge — landed via #107

Thank you — this PR’s intent (analysis pending / claim / release / write-back without opening Mongo as a peer) is on main as part of alabos #107.

What changed vs this branch

  • Routes live under /api/data/analysis/* (not /api/analysis/*), next to samples / measurements / artifacts.
  • AnalysisView and the tests were carried into that unified surface.

Please point desta / analysis clients at the contract in alab_one docs/data-api.md.

— Emma (via Cursor on Sauron)

@EmmaBrass EmmaBrass closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants