Skip to content

Commit 5950d2f

Browse files
docs(observability): clarify runtime log sources and retention limits
Remove Docker copy guidance for the live supervisor tmpfs mount while retaining the Podman example. Distinguish sandbox-runtime security events from supervisor JSONL and gateway log streams, and qualify independent three-file retention. Validation: mise run docs, Markdown lint, and git diff --check passed. Fern reported the same three existing unrelated warnings. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent e06bb7a commit 5950d2f

3 files changed

Lines changed: 19 additions & 11 deletions

File tree

‎docs/observability/accessing-logs.mdx‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ description: "How to view sandbox logs through the CLI, TUI, supervisor files, a
66
keywords: "Generative AI, Cybersecurity, Logging, CLI, TUI, Observability"
77
---
88

9-
Use the CLI or TUI for recent supervisor logs, and compute-driver tooling for supervisor files and main process output. The supervisor runs separately from the sandbox workload, so these logs have different locations and collection paths.
9+
Use the CLI or TUI for recent supervisor logs, and compute-driver tooling for supervisor files, sandbox-runtime events, and main process output. The supervisor runs separately from the sandbox workload, so these logs have different locations and collection paths.
1010

1111
## CLI
1212

@@ -35,14 +35,16 @@ The TUI dashboard displays sandbox logs in real time. Logs appear in the log pan
3535

3636
## Main Process Output
3737

38-
The sandbox's main process writes its stdout and stderr to the sandbox container's stdout and stderr, alongside warnings from the sandbox runtime. Read it with the container tooling for the compute driver:
38+
The sandbox's main process writes its stdout and stderr to the sandbox container's stdout and stderr, alongside sandbox-runtime diagnostics. Read this output with the container tooling for the compute driver:
3939

4040
```shell
4141
kubectl -n <namespace> logs <sandbox-pod> -c agent
4242
docker logs <sandbox-container>
4343
podman logs <sandbox-container>
4444
```
4545

46+
The sandbox runtime also emits OCSF shorthand events, including process-launch and Landlock filesystem events, into these container logs when its diagnostic log filter permits them. These runtime-originated events are not pushed to the gateway log stream or included in the supervisor's JSONL file. Collect workload container logs separately from supervisor logs to retain both sources.
47+
4648
A main process started with a TTY writes only to its attachment; use `openshell sandbox connect` to view that output. Output from `sandbox exec` and SSH sessions does not reach the container log, and `openshell logs` does not include main process output.
4749

4850
When the container runtime falls behind reading the log, the main process blocks on writes, as with any container.
@@ -51,7 +53,7 @@ When the container runtime falls behind reading the log, the main process blocks
5153

5254
The supervisor pushes logs to the gateway over gRPC in real time. The gateway stores up to 2,000 recent log lines per sandbox in memory. This buffer is not persisted to disk and is lost when the gateway restarts or the sandbox's buffers are removed. The push channel is bounded and can drop events under load or during disconnection.
5355

54-
For durable retention, collect supervisor output or files into an external log store. The local files rotate and use temporary storage on container drivers; they are not a durable archive. See [Direct Filesystem Access](#direct-filesystem-access) for locations and collection limits, and [OCSF JSON export](/observability/ocsf-json-export) for structured file output.
56+
For durable retention, collect supervisor output or files and workload container logs into an external log store. The local files rotate and use temporary storage on container drivers; they are not a durable archive. See [Direct Filesystem Access](#direct-filesystem-access) for locations and collection limits, and [OCSF JSON export](/observability/ocsf-json-export) for structured file output.
5557

5658
The gateway's separate `ocsf_log` configuration writes gateway-produced OCSF events to JSONL. It does not persist supervisor log lines received over gRPC or main process output.
5759

@@ -85,15 +87,15 @@ The supervisor writes shorthand logs to `/var/log/openshell.YYYY-MM-DD.log` and,
8587
| VM | `/var/log` on the host running the supervisor, if writable | Host-managed storage. The driver also captures stderr in `supervisor.err.log` in the sandbox's driver state directory. |
8688
| Windows MXC | Gateway-local JSONL output | See [Windows MXC gateways](/observability/ocsf-json-export#windows-mxc-gateways). |
8789

88-
Use platform tooling with access to the supervisor resources. For example, copy the current Docker or Podman supervisor files to a local directory while the container is running:
90+
Use platform tooling with access to the supervisor resources. For Podman, copy the current supervisor files to a local directory while the container is running:
8991

9092
```shell
9193
mkdir -p ./supervisor-logs
92-
docker cp <supervisor-container>:/var/log/. ./supervisor-logs/
93-
# For Podman, use this instead.
9494
podman cp <supervisor-container>:/var/log/. ./supervisor-logs/
9595
```
9696

97+
Docker's `docker cp` cannot copy the live contents of the supervisor's tmpfs log mount. Retrieving those files requires operator-managed access to the live mount on the Docker daemon host. The default supervisor image has no shell or `tar`, so shell-based `docker exec` collection is not an alternative.
98+
9799
For live shorthand output, read the supervisor's stderr through the platform log interface:
98100

99101
```shell
@@ -102,7 +104,7 @@ docker logs --follow <supervisor-container>
102104
podman logs --follow <supervisor-container>
103105
```
104106

105-
This console output follows the supervisor's diagnostic log filter and does not include full OCSF JSON records. A cluster or container log collector can ship it for retention. Collect workload container logs separately for main process output.
107+
This console output follows the supervisor's diagnostic log filter and does not include full OCSF JSON records. A cluster or container log collector can ship it for retention. Collect workload container logs separately for main process output and sandbox-runtime events.
106108

107109
The default supervisor image has no shell or `tar`, so shell-based `kubectl exec` and `kubectl cp` examples do not work with that image. Reading Kubernetes supervisor files requires operator-managed access to its log volume, such as cluster debugging tooling. OpenShell does not configure a file collector for you.
108110

‎docs/observability/logging.mdx‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,8 @@ For L7 REST policy denials, the body also includes structured policy fields such
236236

237237
Landlock filesystem restrictions emit `CONFIG:` events at startup and whenever the sandbox has to skip a requested path.
238238

239+
These events originate in the sandbox runtime and appear in [workload container logs](/observability/accessing-logs#main-process-output) when the runtime's diagnostic log filter permits them. They are not included in the supervisor's JSONL file or pushed to the gateway log stream.
240+
239241
On startup, the probe reports the kernel's supported Landlock ABI version alongside the requested path counts:
240242

241243
```text
@@ -258,10 +260,12 @@ The supervisor writes logs to `/var/log/` in its own filesystem, separate from t
258260

259261
| File | Format | Rotation |
260262
|---|---|---|
261-
| `openshell.YYYY-MM-DD.log` | Shorthand + standard tracing | Daily, 3 files max |
262-
| `openshell-ocsf.YYYY-MM-DD.log` | OCSF JSONL when enabled | Daily, 3 files max |
263+
| `openshell.YYYY-MM-DD.log` | Shorthand + standard tracing | Daily |
264+
| `openshell-ocsf.YYYY-MM-DD.log` | OCSF JSONL when enabled | Daily |
265+
266+
Each supervisor file appender is configured with a three-file limit. The shorthand appender's filename prefix also matches JSONL files, so its cleanup can delete JSONL history on filesystems that expose file creation timestamps. Do not rely on independent three-file retention for each format.
263267

264-
Both files rotate daily and retain the 3 most recent files to bound disk usage. File output requires a writable `/var/log` and uses non-blocking writers that can drop events under pressure. If file logging cannot start, the supervisor falls back to stderr-only output.
268+
File output requires a writable `/var/log` and uses non-blocking writers that can drop events under pressure. If file logging cannot start, the supervisor falls back to stderr-only output.
265269

266270
Docker and Podman use tmpfs for this directory; Kubernetes uses an `emptyDir` in the supervisor pod. Collect logs externally before rotation or teardown for durable retention. See [Accessing Logs](/observability/accessing-logs#direct-filesystem-access) for driver-specific access and storage lifetimes.
267271

‎docs/observability/ocsf-json-export.mdx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,9 @@ do not suppress records in this explicitly enabled audit sink.
7171

7272
For sandbox supervisors, OCSF JSON records are written to `/var/log/openshell-ocsf.YYYY-MM-DD.log` in the supervisor's filesystem, separate from the workload. `openshell sandbox exec` cannot read those files. File output requires a writable `/var/log`; if the supervisor falls back to stderr-only logging, enabling `ocsf_json_enabled` does not create a JSONL sink.
7373

74-
Supervisor files rotate daily and retain the three most recent files. Docker and Podman store them on tmpfs; Kubernetes stores them in the supervisor pod's `emptyDir`. For durable retention, arrange external collection before rotation or teardown. See [supervisor file access](/observability/accessing-logs#direct-filesystem-access) for collection paths and best-effort limits.
74+
This file contains supervisor-originated events, not the [sandbox-runtime events](/observability/accessing-logs#main-process-output) emitted into workload container logs.
75+
76+
Supervisor files rotate daily with a configured three-file limit, but cleanup can remove JSONL history earlier. See [supervisor file retention](/observability/logging#log-file-location) for the limits. Docker and Podman store them on tmpfs; Kubernetes stores them in the supervisor pod's `emptyDir`. For durable retention, arrange external collection before rotation or teardown. See [supervisor file access](/observability/accessing-logs#direct-filesystem-access) for collection paths and best-effort limits.
7577

7678
Windows MXC gateway records use `%PROGRAMDATA%\OpenShell\logs` or `OPENSHELL_OCSF_LOG_DIR`. That sink also rotates daily and retains the three most recent files. The separate `[openshell.gateway.ocsf_log]` sink uses its configured path and retention.
7779

0 commit comments

Comments
 (0)