Skip to content

feat: [OCISDEV-900] Connect storageprovider with coordinator instead of driver - #721

Open
LarsJurgensen wants to merge 9 commits into
mainfrom
OCISDEV-900-pr7-switchover
Open

LarsJurgensen wants to merge 9 commits into
mainfrom
OCISDEV-900-pr7-switchover

Conversation

@LarsJurgensen

@LarsJurgensen LarsJurgensen commented Aug 19, 2026

Copy link
Copy Markdown

Wire storageprovider and dataprovider to the upload coordinator. This is the final step that makes TUS, postprocessing, AV scanning, and session management driver-independent.

  • storageprovider.InitiateFileUpload routes through coord.InitiateUpload instead of fs.InitiateUpload
  • Dataprovider starts the postprocessing NATS consumer on the coordinator, not on the driver
  • datatx.DataTX.Handler gains a coord argument alongside the driver; all three implementations (simple, spaces, tus) updated
  • PUT uploads go through coord.Upload; TUS registers coord.UseIn instead of the driver's UseIn
  • ComposableFS check in the TUS manager removed; the coordinator satisfies the tusd data store interface for every driver
  • Decomposedfs no longer subscribes to PostprocessingFinished, PostprocessingStepFinished, RestartPostprocessing, CleanUpload; the coordinator owns those
  • Decomposedfs keeps RevertRevision under a separate consumer group (<group>-revisions) so it gets its own copy without contending with the coordinator's group
  • Both services gain an upload_directory config key; required for drivers with no local filesystem root

@kw-security

kw-security commented Aug 19, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@LarsJurgensen
LarsJurgensen force-pushed the OCISDEV-900-pr7-switchover branch from b1626ed to 6b608d1 Compare August 19, 2026 14:12
@LarsJurgensen LarsJurgensen changed the title Ocisdev 900 pr7 switchover feat: [OCISDEV-900] Connect storageprovider with coordinator instead of driver Aug 20, 2026
@LarsJurgensen
LarsJurgensen marked this pull request as ready for review August 20, 2026 14:13
@LarsJurgensen
LarsJurgensen requested a review from a team as a code owner August 20, 2026 14:13

@LukasHirt LukasHirt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two correctness regressions, one reachability concern, two cleanups.

Blocking:

  1. TouchFile's create-only symlink leaks an orphaned node on a concurrent-create race (pkg/storage/utils/decomposedfs/tree/tree.go:175).
  2. OC-FileId is now permanently omitted from TUS responses for newly-created files (pkg/rhttp/datatx/manager/tus/tus.go:216).

Worth a look:
3. This PR is what makes pkg/upload's postprocessing path live for the dataprovider — and that path has no guard against a node vanishing mid-postprocessing (internal/http/services/dataprovider/dataprovider.go:117).

Minor:
4. Doubled NATS subscriptions for combined decomposedfs+dataprovider deployments (same line).
5. getCoordinator duplicated near-verbatim with storageprovider.go:234 (internal/http/services/dataprovider/dataprovider.go:146).

Comment thread pkg/storage/utils/decomposedfs/tree/tree.go
Comment thread pkg/rhttp/datatx/manager/tus/tus.go
Comment thread internal/http/services/dataprovider/dataprovider.go
Comment thread internal/http/services/dataprovider/dataprovider.go Outdated
@LarsJurgensen
LarsJurgensen force-pushed the OCISDEV-900-pr7-switchover branch from 79bb052 to 4717d97 Compare September 9, 2026 14:31
@LarsJurgensen
LarsJurgensen force-pushed the OCISDEV-900-pr7-switchover branch from 4717d97 to d8ba908 Compare September 16, 2026 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants