|
| 1 | +# Scoped host resources and the host extension SDK |
| 2 | + |
| 3 | +RustScript's `pd-vm` core is **host-agnostic**: `src/vm`, the generic resource/operation cores and |
| 4 | +`ExecutionScope` never import or dispatch a concrete host library (`rusqlite`, `hyper`, `tokio::net`, |
| 5 | +`tokio::process`, platform process/thread implementations). Concrete capabilities — SQLite, file/socket/ |
| 6 | +process I/O, HTTP/SSE — are supplied by *same-crate standard builtins* (and by external host crates) |
| 7 | +that consume the generic scoped host SDK documented here. |
| 8 | + |
| 9 | +The architecture is a Deno/Wasmtime-style hybrid: |
| 10 | + |
| 11 | +- the core owns an object-safe [`HostResource`] interface and a typed, generational |
| 12 | + [`ResourceTable`] of erased resources; |
| 13 | +- host extensions register arbitrary concrete resources and dynamic [`HostOperation`] drivers; |
| 14 | +- one [`ExecutionScope`] binds the resource table and operation registry to a single VM invocation; |
| 15 | +- `Vm` reset closes the old scope and builds a fresh guest execution state; it never queries host |
| 16 | + module history, resource classes or host-function history. |
| 17 | + |
| 18 | +## `HostResource` implementation guide |
| 19 | + |
| 20 | +A concrete resource implements the object-safe trait: |
| 21 | + |
| 22 | +```rust |
| 23 | +use vm::{CloseProgress, HostResource, ResourceCloseReason, ResourceResult}; |
| 24 | + |
| 25 | +struct MyResource { /* owned native state */ } |
| 26 | + |
| 27 | +impl HostResource for MyResource { |
| 28 | + fn begin_close(&mut self, reason: ResourceCloseReason) -> ResourceResult<CloseProgress> { |
| 29 | + // 1. Synchronously issue any cancel/close request to the underlying |
| 30 | + // work (interrupt a query, signal a thread, close a socket). |
| 31 | + // 2. Return CloseProgress::Ready if nothing remains, or |
| 32 | + // CloseProgress::Pending to drive poll_close afterwards. |
| 33 | + Ok(CloseProgress::Ready) |
| 34 | + } |
| 35 | + |
| 36 | + fn poll_close(&mut self, cx: &mut std::task::Context<'_>) -> std::task::Poll<ResourceResult<()>> { |
| 37 | + // Only called after begin_close returned Pending. Drive the close to |
| 38 | + // completion; return Poll::Pending and register cx.waker() if the |
| 39 | + // underlying work is still running. |
| 40 | + std::task::Poll::Ready(Ok(())) |
| 41 | + } |
| 42 | +} |
| 43 | +``` |
| 44 | + |
| 45 | +Contract rules: |
| 46 | + |
| 47 | +- `begin_close` **must be idempotent** and must synchronously issue the cancel/close request. A |
| 48 | + resource that needs asynchronous teardown returns `Pending` and completes it in `poll_close`. |
| 49 | +- `poll_close` is called only after `begin_close` returned `Pending`. |
| 50 | +- A concrete `Drop` remains the **last-resort guard** for memory/OS-handle safety, but the VM may |
| 51 | + only reuse a resource (and its slot) once `poll_close` completes. |
| 52 | +- Resources should override `resource_type_key()` with their stable catalog key (for example |
| 53 | + `"sqlite.connection"`) so exact host-import schemas can validate them; the default returns `None` |
| 54 | + (legacy typed APIs only). |
| 55 | +- The core records only *generic* close errors; concrete host crates own error mapping and |
| 56 | + diagnostics. |
| 57 | + |
| 58 | +The `HostResource` bound is `Any + Send + 'static`. A resource must be `Send` because the table |
| 59 | +(and thus the `Vm`) is `Send`; it is deliberately **`!Sync`** — the table is owned and mutated by a |
| 60 | +single thread, and Rust references are only borrowed for the duration of one host call. |
| 61 | + |
| 62 | +## Typed vs raw handles |
| 63 | + |
| 64 | +- A [`Resource<T>`] is the **typed host-side token**: a `Copy` capability keyed by a |
| 65 | + [`ResourceHandle`]. Duplicating the token does **not** duplicate ownership of the underlying |
| 66 | + resource. |
| 67 | +- A [`ResourceHandle`] is the **raw guest-facing token**: an opaque integer that carries only |
| 68 | + `arena/scope identity | slot index | generation`. It can cross the host boundary as a script |
| 69 | + `Value::Int`, but it encodes **no** domain resource type. |
| 70 | +- The table validates a typed access in order: handle encoding → arena/scope identity → slot index |
| 71 | + and generation → slot state `Open` → slot `TypeId` equals `TypeId::of::<T>()` → ownership |
| 72 | + transition. Passing a `Resource<File>` where a `Resource<SqliteConnection>` is expected returns a |
| 73 | + typed `ResourceTypeMismatch` and leaves the original resource untouched (no ownership consumed, |
| 74 | + no borrow changed, no cleanup run, no generation advanced). |
| 75 | +- Raw handles are valid only within the current VM execution scope. They must never be persisted or |
| 76 | + shared across VMs; an old-scope handle is rejected with `ResourceHandleWrongTable` and a reused |
| 77 | + slot with a stale generation is rejected with `ResourceStale`. |
| 78 | + |
| 79 | +Host functions borrow a resource for the duration of one call through `ResourceTable::get` / |
| 80 | +`get_mut`, returning `ResourceRef<'a, T>` / `ResourceMut<'a, T>`. **Rust borrows never outlive a |
| 81 | +yield or a pending operation**; asynchronous work must hold its own state or a `Resource<T>` handle. |
| 82 | + |
| 83 | +## Parent/child rules |
| 84 | + |
| 85 | +Resources can be registered as children of a parent: |
| 86 | + |
| 87 | +- `push_child_resource::<T, P>(value, &parent)` links `T` under an open `P`; the parent cannot be |
| 88 | + closed while the child is live. |
| 89 | +- Explicit single-resource close of a parent with live children returns |
| 90 | + `ResourceHasChildren`. |
| 91 | +- **Scope shutdown uses a deterministic post-order (child-first) order**: every leaf is begun, then |
| 92 | + its parent once the child has completed. This guarantees a Pending child can never prevent its |
| 93 | + parent's `begin_close` from running before the owning tables fall through to their `Drop` guards. |
| 94 | +- Closing a resource cancels operations associated with that exact handle (generic association; the |
| 95 | + core never dispatches on a resource class). |
| 96 | + |
| 97 | +## Scope ownership, reset and the Vm Drop contract |
| 98 | + |
| 99 | +Every `Vm` owns exactly one `ExecutionScope` (resource table + operation registry + close state). |
| 100 | +`HostContext` (obtained via `vm.host_context()`) is the guarded mutation surface: it pushes |
| 101 | +resources, starts operations, borrows/validates typed resources, installs module state and begins |
| 102 | +single-resource closes — without ever exposing `HostRuntime` private fields. |
| 103 | + |
| 104 | +Reuse is an explicit two-phase contract: |
| 105 | + |
| 106 | +- `Vm::begin_reset_for_reuse(reason, deadline)` begins scope shutdown (Active → Closing, sealing new |
| 107 | + inserts). First reason/deadline wins; repeated begins are idempotent. |
| 108 | +- `Vm::poll_reset_for_reuse(cx, now)` drives the close to quiescence. Only when the scope is |
| 109 | + `Quiescent` (operations drained, resources closed) is a fresh `Active` scope installed and the |
| 110 | + guest execution state rewound. While pending, the VM is `Resetting` and never lent out of a pool. |
| 111 | +- `Vm::reset_for_reuse()` is the synchronous compat entry; with genuinely pending resources it |
| 112 | + returns a structured `ResetPending` and the VM stays `Resetting` until driven through the poll API. |
| 113 | + It never busy-loops. |
| 114 | + |
| 115 | +**`Vm` Drop** (plan section 5.3): dropping a `Vm` synchronously begins the execution-scope close |
| 116 | +with `ResourceCloseReason::VmDrop` and drives one round of the close pipeline with a no-op waker — |
| 117 | +cancelling every pending operation with `OperationCancelReason::VmDrop` and issuing child-first |
| 118 | +`begin_close` to every live resource with `ResourceCloseReason::VmDrop`. Drop never blocks, never |
| 119 | +claims quiescence and never recycles; genuinely event-driven `Pending` resources stay `Closing` and |
| 120 | +are released by their own `Drop` guards. Guest-owned local handles are released (exactly-once) with |
| 121 | +the ownership-release reason before the scope shutdown, and scope shutdown closes anything that |
| 122 | +survived. |
| 123 | + |
| 124 | +Module policy (e.g. `SqlitePolicy`, `IoPolicy`, `HttpConfig`) lives in **persistent per-VM module |
| 125 | +state** ([`HostModuleState`]): it survives scope close and reset, never participates in resource |
| 126 | +close, and is keyed by `TypeId`. |
| 127 | + |
| 128 | +## Synchronous and asynchronous close examples |
| 129 | + |
| 130 | +Synchronous close (a resource that tears down inline): |
| 131 | + |
| 132 | +```rust |
| 133 | +// Scope shutdown calls begin_close on every live resource (child first) with |
| 134 | +// the shutdown reason; a Ready resource is reclaimed immediately. |
| 135 | +``` |
| 136 | + |
| 137 | +Asynchronous close (a cooperative worker thread): |
| 138 | + |
| 139 | +```rust |
| 140 | +impl HostResource for WorkerResource { |
| 141 | + fn begin_close(&mut self, _reason: ResourceCloseReason) -> ResourceResult<CloseProgress> { |
| 142 | + self.cancelled.store(true, Ordering::SeqCst); // cooperative cancel |
| 143 | + Ok(CloseProgress::Pending) |
| 144 | + } |
| 145 | + fn poll_close(&mut self, cx: &mut Context<'_>) -> Poll<ResourceResult<()>> { |
| 146 | + if self.join_done() { |
| 147 | + Poll::Ready(Ok(())) |
| 148 | + } else { |
| 149 | + cx.waker().wake_by_ref(); // re-poll when progress is possible |
| 150 | + Poll::Pending |
| 151 | + } |
| 152 | + } |
| 153 | +} |
| 154 | +``` |
| 155 | + |
| 156 | +During a reset, `poll_reset_for_reuse` re-polls pending resources with the caller's waker until |
| 157 | +quiescence (or the recycle deadline). The core never force-kills a thread; a worker that does not |
| 158 | +join before the host recycle deadline causes the VM to be discarded (poisoned). |
| 159 | + |
| 160 | +## Cleanup failure and the poisoned VM |
| 161 | + |
| 162 | +Shutdown is best-effort: an error on one resource never skips the remaining resources. The terminal |
| 163 | +scope outcome carries the first typed error plus the failure count. |
| 164 | + |
| 165 | +- A **cleanup error** or **recycle deadline** during reset moves the VM to `VmResetState::Poisoned`. |
| 166 | + The old scope (with its recorded error) is preserved for diagnostics; the VM can be dropped but |
| 167 | + never runs again and never returns to a pool. |
| 168 | +- An **explicit single-resource close failure** stays local to that resource: the error is returned |
| 169 | + to the caller, the resource stays open, and scope shutdown retries the idempotent close request. |
| 170 | +- A poisoned VM reports the failure through `Vm::reset_error()` / `Vm::reset_state()` and rejects |
| 171 | + `run`/`resume`/reuse with a structured `NotReusable` error. |
| 172 | + |
| 173 | +## `HostOperation` and `HostContext` usage |
| 174 | + |
| 175 | +A pending host operation is an object-safe driver: |
| 176 | + |
| 177 | +```rust |
| 178 | +use vm::operation::{HostOperation, OperationCancelReason, OperationResult}; |
| 179 | +use std::task::{Context, Poll}; |
| 180 | + |
| 181 | +struct MyOp; |
| 182 | +impl HostOperation for MyOp { |
| 183 | + fn poll(&mut self, cx: &mut Context<'_>) -> Poll<OperationResult<()>> { Poll::Pending } |
| 184 | + fn cancel(&mut self, reason: OperationCancelReason) -> OperationResult<()> { Ok(()) } |
| 185 | +} |
| 186 | +``` |
| 187 | + |
| 188 | +Operations are registered through `HostContext::start_operation(OperationSpec::new(driver))`; a spec |
| 189 | +may carry a deadline, an associated `ResourceHandle` (closing that resource cancels the operation) |
| 190 | +and a one-shot cleanup. Scope shutdown cancels every pending operation with a single typed reason and |
| 191 | +drains the registry. There is no static owner→poller table and no global token tree. |
| 192 | + |
| 193 | +`HostContext` also provides: |
| 194 | + |
| 195 | +- `push_resource` / `push_resource_with_key` / `push_child_resource` — insert resources; |
| 196 | +- `get` / `get_mut` — call-scoped typed borrows; |
| 197 | +- `close_resource::<T>(handle, reason)` — explicit single-resource close; |
| 198 | +- `mark_resource_guest_owned(handle)` — exact host-return ownership transfer; |
| 199 | +- `set_module_state` / `module_state` / `module_state_mut` — persistent typed module state; |
| 200 | +- `execution_scope()` — read-only scope observations (counts, state, terminal outcome). |
| 201 | + |
| 202 | +External host crates compose through the `HostExtension` trait: `register(registry)` registers exact |
| 203 | +host functions from a `HostApiCatalog` (via `catalog_import_schemas`), and `install(vm)` installs |
| 204 | +persistent module state. `Vm::install_extension(&extension)` runs both steps transactionally. The |
| 205 | +exact schemas — parameter labels, type schemas, passing modes and the catalog fingerprint — must |
| 206 | +match byte-for-byte what the compiler embeds in the program's `HostImport`, so a registry compiled |
| 207 | +against a different catalog is rejected at bind time. |
| 208 | + |
| 209 | +## Feature matrix |
| 210 | + |
| 211 | +| Feature set | `src/vm` / resource / operation / scope | Standard builtins | Compiler / catalog | |
| 212 | +|---|---|---|---| |
| 213 | +| `pd-vm --no-default-features` | generic core only; no OS hosts | none | catalog wire types available | |
| 214 | +| `pd-vm --no-default-features --features runtime` | generic core only | `io::*` surface | `HostApiCatalog` snapshot | |
| 215 | +| `+ sqlite` | generic core only (no `cfg(feature = "sqlite")` in `src/vm`) | SQLite builtin (rusqlite, optional dep) | sqlite surface in the catalog | |
| 216 | +| `+ http-client` | generic core only (no `cfg(feature = "http-client")` in `src/vm`) | HTTP/SSE builtin (hyper/rustls) | http surface in the catalog | |
| 217 | +| `pd-vm-nostd` | n/a (no compiler/VM; VMBC v13 decoder only) | none | decodes exact `HostImport` schemas | |
| 218 | +| `pd-vm-wasm` (`runtime` feature) | generic core compiled to wasm32 | io surface when enabled | — | |
| 219 | + |
| 220 | +The `sqlite` / `http-client` features only decide whether the same-crate standard builtin is |
| 221 | +compiled and registered by default; they never enter the resource/reset architecture. `src/vm`, |
| 222 | +`src/vm/resource`, `src/vm/operation` and `ExecutionScope` carry no `cfg(feature = "sqlite")` or |
| 223 | +`cfg(feature = "http-client")` and never import the concrete host libraries. `tests/ |
| 224 | +core_host_boundary_tests.rs` enforces this at the source level. |
| 225 | + |
| 226 | +Enabling a standard extension in an embedding: |
| 227 | + |
| 228 | +```toml |
| 229 | +pd-vm = { git = "https://github.com/rustscript-lang/rustscript", package = "pd-vm", |
| 230 | + features = ["sqlite", "http-client"] } |
| 231 | +``` |
| 232 | + |
| 233 | +then the standard catalog is available through `builtins::runtime::standard_host_catalog()` and the |
| 234 | +standard compile entry installs the same snapshot, so compiled `HostImport` fingerprints match the |
| 235 | +registered exact schemas for any combination of enabled features. |
0 commit comments