Repository navigation
vm: 补齐 PostgreSQL Store 与共享 Store 的「一 Store 一 VM」 - #531
Closed
jiangkoumo wants to merge 4 commits into
Closed
jiangkoumo wants to merge 4 commits into
jiangkoumo wants to merge 4 commits into
Conversation
Cover gap 1 noted in the Ephemeral-AI-Lab#526 merge (81f8758) so projects of a PostgreSQL Store (Ephemeral-AI-Lab#506) work under a `vm:<provider>` home (Ephemeral-AI-Lab#515): - In @layerfs/runtime: - `platform/vm/index.js` `deploymentEnv` returns `STORE_PATH=postgresql:<store>`, `STORE=<store>` and, when `project.url` is given, `STORE_POSTGRESQL={"url":...}`, while keeping the VM's `SANDBOX=native`, `SANDBOX_BACKEND=namespaces`, `DAEMON`, `LOCAL_DIR`, `IMAGE`, `IMAGE_ARCHIVE`, `CONTAINER_PREFIX` and `WORKER_ENDPOINT`. - `ensure`, `init`, `makeStore`, `removeStore` and `vm.delete`: a PostgreSQL project's VM creates and mounts no Store disk (on microsandbox, apple-container and ssh), runs no `mkdir /layerfs/store` or source copy in the VM, and removes no Store disk when the project, Store or VM is deleted. `releaseStore` releases `postgresql:<store>` in the project's worker. - `checkVmUrl`: inside a VM, `localhost`, `127.0.0.1`, `::1` and host socket paths refer to the VM itself rather than the host; PostgreSQL URLs naming loopback or a host socket on a `vm:` home are refused `InvalidInput` before any worker or daemon call, without including the URL's credentials in the error message. - `project/delete.js` `dropStack`: on a `vm:` home (`hostSandboxes === false`), checks and closes running Sandboxes through `storeHolders`/`releaseStoreHolders` instead of host Docker `containers()`, and removes the project's VM after dropping its stack. - In layerfs-api, layerfs-node and layerfs-sandbox: - Open up `local::start` and `SandboxApi::create_provider_postgres` for `DaemonStore::Postgres` on native/namespaces deployments: a PostgreSQL Sandbox sets `setup.postgresql` and passes `/dev/null` as the namespaces Store path so `namespaces::create` and `layerfs-sandbox-helper` bind-mount no Store directory into the Sandbox's private root. - Tests and docs: - Add `vm_placement.mjs` `postgresql` selection (plus `flow` and `apple-container` checks) and a `provider_ownership` test in `layerfs-api`. - Document `vm:` PostgreSQL Store rules in `api/docs/runtime.md`. Verified on macOS 27.2 ARM64 (Rust 1.85.1, locked): - `cargo +1.85.1 clippy --locked --workspace --all-targets -- -D warnings`: exit 0 - `cargo +1.85.1 clippy --locked -p layerfs-node --all-targets -- -D warnings`: exit 0 - `cargo +1.85.1 fmt --all -- --check`: only the pre-existing minio.rs drift - `cargo +1.85.1 test --locked -p layerfs-sandbox`: 56 passed, 0 failed, 1 ignored - `cargo +1.85.1 test --locked -p layerfs-api --test provider_ownership`: 1 passed - `node api/runtime/test/runtime.mjs`: all 15 selections PASS - `node api/runtime/test/vm_placement.mjs`: all 10 selections PASS - `python3 -B tools/check_product_boundary.py`: PASS (1119 files) - `python3 -B -m unittest discover -s tools -p 'test_*.py'`: 156 tests OK Not run: real PostgreSQL server and real VM E2E (no Docker/PostgreSQL running on the host). Disposable only; Durable NOT_RUN, disabled by owner. Production LOC: 123141 -> 123235 (delta +94) First-parent comparison: layerfs_v2_0_0 81f8758 (tree 01a2878) to committed tree 140fe6a. Scopes: core 41405 -> 41405 (+0), engine 77512 -> 77524 (+12), api 4224 -> 4306 (+82), dsh 0 -> 0 (+0). By crate: @layerfs/runtime 2649 -> 2731 (+82), layerfs-node 5749 -> 5752 (+3), layerfs-sandbox 8260 -> 8269 (+9), layerfs-api 2106 -> 2106 (+0). Method: tools/production_loc.py (blob 686d269) --root <exported tree> --json for both snapshots; first-party production Rust, runtime SQL and JavaScript; tests, docs, tooling, fixtures, generated and third-party files excluded. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Deleted tests: none.
Wire shared SQLite Stores (Ephemeral-AI-Lab#506) into vm:<provider> placements (Ephemeral-AI-Lab#515 V1) so each SQLite Store owns one VM, one Store disk and one worker across all its projects: - platform/vm/index.js: resolve a SQLite Store or any project in it to the Store's VM (<prefix><store>-vm), disk (<prefix><store>-vm-store) and worker endpoint (<home>/worker/<store>.json); include LAYERFS_STORE=<store> in deploymentEnv for shared-Store projects; aggregate running Sandboxes across all projects of a Store in storeHolders/releaseStoreHolders; enforce V8 and disk free-space accounting on registered SQLite Stores. - store/calls.js and store/sync.js: provision the Store's VM and disk in makeVolume when hostDocker is false instead of calling host docker volume inspect/create, and dispatch Store backend calls (create, stat, load, publish, push, pull, send, import_postgresql, sync.progress) through that Store's VM worker. - project/init.js and project/delete.js: pass the target project to platform.init in runInto so the source folder is copied into the shared Store's VM and runtime.store.add_project runs in its worker; keep the shared SQLite Store's VM and disk intact when dropStack deletes a single project. - test/vm_placement.mjs and docs/runtime.md: cover shared Store create, load, project init, project delete and store remove across fake-msb, fake-container and fake-ssh fixtures. Production LOC: 123235 -> 123296 (delta +61) First-parent comparison: v2-gap-vm-postgresql c9dc9b2 (tree 140fe6a) to committed tree 487eae9. Scopes: core 41405 -> 41405 (+0), engine 77524 -> 77524 (+0), api 4306 -> 4367 (+61), dsh 0 -> 0 (+0). By crate: @layerfs/runtime 2731 -> 2792 (+61). Method: python3 -B tools/production_loc.py (blob 686d269) --root <exported tree> --json for both snapshots; first-party production Rust, runtime SQL and JavaScript; tests, docs, tooling, generated and third-party files excluded. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Deleted tests: none.
… Local PostgreSQL Sandboxes
- Replace /dev/null sentinel path with store: Option<PathBuf> in
ProviderLaunchOptions::Namespaces, namespaces::create::Options, and
HelperSpec (None omits the store= line and binds no Store directory
for PostgreSQL Stores).
- Restore the Unsupported refusal in layerfs-node/src/sandbox.rs when
a PostgreSQL Store is used with the host Local (Backend::Native) Sandbox
backend, and reject ProviderLaunchOptions::Local in
SandboxApi::create_provider_postgres, allowing only Docker { store_volume: None }
and Namespaces { store: None }.
- Expand provider_ownership and namespaces_protocol tests to cover
Local + PostgreSQL rejection, Namespaces + Some(dir) rejection,
Namespaces + None admission, and HelperSpec optional store encoding.
Production LOC: 123296 -> 123309 (delta +13)
Scope: first-party product source counted by tools/production_loc.py (blob 686d269)
First-parent comparison: eef19e8 -> HEAD
Subtotals: core 41405 -> 41405 (+0), engine 77524 -> 77537 (+13: layerfs-node 5752 -> 5760, layerfs-sandbox 8269 -> 8274, layerfs-api 2106 -> 2106), api 4367 -> 4367 (+0), dsh 0 -> 0 (+0); reference absent (retired R9c; last counted 65,417 at tree 498dd19)
…status - Pass full project objects from project.list() into runtime.status() and runtime.vm.status() so same-name projects across Stores keep their Store identity instead of failing InvalidInput or zeroing status.vm; check same-name project ambiguity in vmKeyOf across solo and shared Stores. - Unify PostgreSQL Stores on vm: placement to one VM and worker endpoint per Store (<prefix><store>-vm, <home>/worker/<store>.json); ensure the Store VM worker before dropStack remove_project and keep the Store VM when removing one project from a PostgreSQL Store. - Route start_import_postgresql list_projects, attach_postgresql, store.check and store.unlink through the Store VM worker after ensuring it runs; clean up .stores/<name> and remote-state/<name> on store.remove; ensure VM workers before sync jobs and reject empty/unspecified hosts in checkVmUrl. Production LOC: 123309 -> 123335 (delta +26) - First-parent comparison: 685d71f (tree 904410e) to committed tree - Scopes: core 41405 -> 41405 (+0), engine 77537 -> 77537 (+0), api 4367 -> 4393 (+26), dsh 0 -> 0 (+0) - By crate: @layerfs/runtime 2792 -> 2818 (+26) - Method: python3 -B tools/production_loc.py --root . --json (blob 686d269) - Reference: reference absent (retired R9c; last counted 65,417 at tree 498dd19)
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
37 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
#526 合入
layerfs_v2_0_0时记下了vm:部署的两处缺口:vm:home 的deploymentEnv不覆盖 PostgreSQL Store;store.create/load仍在宿主机建 Docker 卷,没有按 docs(512): propose a VM Sandbox backend (Sandboxes as namespaces in one VM per Store) #515「一个 Store 一台 VM」走。本 PR 补齐这两处,目标分支
layerfs_v2_0_0(基线 81f8758)。改了什么
PostgreSQL Store 接入
vm:deploymentEnv为 PostgreSQL Store 注入连接信息与WORKER_ENDPOINT;VM 按 Store 名命名(<prefix><store>-vm),同一 PostgreSQL Store 下的所有项目共用一台无盘 VM。platform.init再删项目;store.remove时才销毁 VM。共享 SQLite Store:一个 Store 一台 VM
vm:下的调用都走 Store VMstart_import_postgresql(list_projects)、attach_postgresql、store.check、store.unlink、store.remove(forget)改走storeCall/platform.init,不再在宿主机裸调;removeStore同时清理 VM 内和宿主 home 下的.stores/<name>、remote-state/<name>。start_*前确保 VM worker 已启动;checkVmUrl拒绝空 host、0.0.0.0、::。跨 Store 同名项目
runtime.vm.status()/runtime.status()保留项目的store字段,不再因同名项目抛InvalidInput或返回vm: null。vmKeyOf在未指定 store 时汇总独立项目与所有 Store 的同名项目,多于一个时报InvalidInput: ... name its { store }。引擎层(Rust)
"/dev/null"哨兵改为Option<PathBuf>;namespaces +None允许,namespaces +Some(dir)拒绝。Unsupported(保持 B1 既有决定)。provider_ownership、namespaces_protocol单测。怎么验证
Node 侧:
api/runtime/test/vm_placement.mjs、runtime.mjs全部通过,新增用例覆盖 PostgreSQL 一 Store 一 VM、共享 Store 路由、跨 Store 同名 status 与歧义报错。Linux(三台机器,与基线 81f8758 同机对比失败集合):
cargo clippy --workspace --all-targets -- -D warningscargo fmt --checklayerfs-remote-storage/tests/minio.rs:10既有漂移cargo test --workspacecargo test -p layerfs-api -p layerfs-daemon -p layerfs-sandboxstalled_unmount的AlreadyExists是并行夹具竞争,基线同样出现风险
vm:部署路径;宿主 Docker、原生 Local/namespaces 的行为不变(Local + PostgreSQL 仍为Unsupported)。stalled_unmount有既有偶发竞争,与本 PR 无关。