This repository has grown from a format implementation into a complete TOON toolkit. Use this map to find each user-facing area; links point to the relevant documentation or source directory, depending on the component's maturity.
| Area | What is here |
|---|---|
| Formats and specifications | The pinned official TOON v4.1.1 baseline, RedDB opt-in extensions, and the TOONL streaming specification. |
| Codec libraries and streaming | The @reddb-io/toon package and reddb-io-toon crate: codecs, event streams, truncation reports, TOONL readers/writers, and JSON bridges. See What ships. |
| RPC family | The quarantined draft TOON-RPC protocol, experimental libraries and transports, JSON-RPC/TOON-RPC negotiation, MCP/legacy ACP adapters, prototype code generation and CLI tooling, and examples. Kept together under RPC family. |
| Command-line tools | The drop-in toon converter and the tq query/conversion CLI for TOON, TOONL, JSON, YAML, and XML, with its language reference and jq parity record. |
| Editor integration | The RedDB Toon VS Code extension for .toon, .toonl, and fenced Markdown blocks. |
| Benchmarks and evidence | Reproducible accuracy, token-efficiency, and runtime benchmarks, shared conformance, parity, golden, and adversarial test corpora, and pinned specification and reference implementation checkpoints. |
| Examples | RPC clients and servers, codec examples, and editor grammar samples. |
| Design and migration records | The v4.1 migration guide, design-history proposals, upstream monitoring, and feedback ledger. |
| Install, releases, and development | Prebuilt binaries, the install.sh installer, release history, GitHub releases, and the development workflow. |
TOON was created by Johann Schopplich. The official toon-format project and toonformat.dev are the normative authorities for the format.
This repository is RedDB's practical laboratory. It follows the official format while exploring and shipping the libraries, command-line tools, streaming support, and opt-in extensions needed by RedDB's own production workflows. It is not the original project and is not endorsed by upstream.
TOON is a token-oriented object notation for carrying structured JSON-shaped data through prompts and pipelines with less syntax overhead. It keeps the JSON data model, adds length-bearing tabular forms, and makes common truncation failures visible to decoders.
TOONL is the append-only stream form: one record per line, header once, and optional trailers for closed-stream verification. It is the streaming layer used by the JS package, Rust crate, and tq.
The root README is a hub, not the normative spec. Use these documents for detail:
This repository pins the official TOON v4.1.1 specification as its baseline and layers a set of opt-in extensions on top of it.
- Official TOON baseline: the pinned upstream release, API boundaries, frontier status, and executable evidence.
- RedDB TOON extensions: userland features layered on v4.1, with each decode/encode opt-in and fallback rule stated explicitly.
- TOONL RedDB spec: append-only stream grammar and reader/writer behavior.
- v4.1 migration notes: TypeScript and Rust cutovers from the retired pre-v4 baseline, with observable before/after behavior.
- Design-history proposals: the design history behind each extension — including the mechanisms the official spec absorbed at v4.1.
toonis the drop-in converter. Its TypeScript and Rust front ends are compatible with the pinned upstream v4.1.1 package and CLI contract.tqis the advanced jq-style query and transformation tool. It reads TOON, JSON, YAML, XML, and TOONL and can emit TOON, JSON, XML, or TOONL.
Compatibility here is deliberately scoped and executable. The toon implementations pass the vendored upstream package and CLI compatibility gates, share byte-for-byte JavaScript/Rust CLI goldens, and are checked for Rust↔TypeScript encoder parity against the pinned v4.1.1 baseline. See the pinned compatibility record and options inventory.
tq does not claim universal jq equivalence. Its relationship to jq 1.7.1 is verified by a vendored parity corpus, a documented divergence ledger, and tq jq-check, which emits a machine-readable compatibility classification for a particular filter and option set.
Dependency-free ESM for applications that need TOON in JavaScript, TypeScript, Node, Bun, Deno, or browsers. It parses TOON into plain JSON-shaped values, serializes those values back to canonical TOON, detects common truncation failures before a partial model response is trusted, and includes TOONL helpers for append-only record streams.
Use it when a prompt or pipeline wants compact structured data but the application still needs normal JSON objects at the edges.
pnpm add @reddb-io/toonimport { decode, encode } from '@reddb-io/toon'
const document = decode('users[2]{id,name}:\n 1,Ada\n 2,Linus\n')
console.log(document.users[0].name)
process.stdout.write(`${encode(document)}\n`)Adausers[2]{id,name}:
1,Ada
2,Linusimport { encodeRecords, parseRecords } from '@reddb-io/toon'
const stream = encodeRecords([
{ id: 1, name: 'Ada' },
{ id: 2, name: 'Linus' },
])
process.stdout.write(stream)
console.log(JSON.stringify(parseRecords(stream)))[]{id,name}:
1,Ada
2,Linus
[=2][{"id":1,"name":"Ada"},{"id":2,"name":"Linus"}]Check completeness before accepting generated or streamed data:
import { detectTruncation } from '@reddb-io/toon'
const report = detectTruncation('items[2]:\n - one\n')
console.log(report.complete)
console.log(report.kind)falsearray_length_mismatchDetails: packages/toon, TOON spec companion, RedDB TOON extensions, TOONL spec, and the truncation report model.
TOON-RPC is a draft transport-independent protocol with JSON-RPC semantics and TOON serialization. The entire RPC family is currently quarantined under the 0.30 recovery: no listed surface is production-ready, and automatic npm/crates.io publication is paused until the protocol, interoperability, lifecycle, and package gates pass.
| Surface | Contents |
|---|---|
| Protocol | TOON-RPC specification and JSON-RPC/TOON-RPC wire negotiation |
| TypeScript prototypes | @reddb-io/toon-rpc client/server, @reddb-io/multi-rpc multi-protocol dispatcher, and quarantined MCP and legacy ACP adapters |
| Rust prototypes | reddb-io-toon-rpc, stdio, HTTP, SSE, TCP, WebSocket, and long polling |
| Tooling prototypes | IDL code generation, RPC CLI, quarantined MCP, legacy ACP, and examples |
The source owner of MultiRpc remains packages/multi-rpc; it is not a
subpath of packages/toon-rpc.
The Rust library behind the CLIs and a standalone crate for services that want TOON without shelling out. It provides the parser, serializer, ordered document model, event decoder, truncation detector, JSON bridges, and TOONL reader/writer utilities used by toon and tq.
Use it for Rust pipelines that need canonical TOON output, bounded parsing for untrusted input, or append-only TOONL streams that can be checked and resumed.
cargo add reddb-io-toonuse reddb_io_toon::Value;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let document = Value::parse_toon("users[2]{id,name}:\n 1,Ada\n 2,Linus\n")?;
println!("{}", document.to_canonical_toon());
Ok(())
}Detect a truncated TOON document without losing the structured reason:
use reddb_io_toon::detect_truncation;
fn main() {
let report = detect_truncation("items[2]:\n - one\n");
assert!(!report.complete);
assert_eq!(report.to_json_value()["kind"], "array_length_mismatch");
}Write and read a small TOONL stream:
use reddb_io_toon::{encode_toonl_values, ToonlReader, Value};
use std::io::Cursor;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let rows = vec![
Value::from_json_str(r#"{"id":1,"name":"Ada"}"#)?,
Value::from_json_str(r#"{"id":2,"name":"Linus"}"#)?,
];
let stream = encode_toonl_values(&rows)?;
let decoded = ToonlReader::new(Cursor::new(stream.as_bytes()))
.collect::<Result<Vec<_>, _>>()?;
assert_eq!(decoded, rows);
Ok(())
}Details: crates/toon, decoder and encoder options, RedDB extension rules, TOONL streaming format, and the truncation report model.
An advanced jq-style command-line tool for querying and transforming data at the terminal. It queries TOON, JSON, YAML, XML, and TOONL rows; converts between TOON, TOONL, JSON, and XML; checks TOON or TOONL for truncation; and closes or trims append-only streams.
The shipped language includes def functions with closures and bounded recursion, paths and recursive descent, assignments, string interpolation and @formats such as @json and @csv, and --arg/--argjson bindings. The language catalog marks every builtin as supported, deferred, or never; the jq parity record above documents the precise compatibility boundary.
Use it in shell pipelines to query a model response, turn JSON/YAML into compact TOON for a prompt, convert record streams to TOONL, or verify that a stream ended cleanly.
curl -fsSL https://raw.githubusercontent.com/reddb-io/toon/main/install.sh | shprintf 'users[2]{id,name}:\n 1,Ada\n 2,Linus\n' \
| tq '.users[].name'Convert JSON records into TOONL for append-only logs:
printf '{"id":1,"name":"Ada"}\n{"id":2,"name":"Linus"}\n' \
| tq -p json -o toonl .Query YAML input and emit compact JSON:
printf 'users:\n - id: 1\n name: Ada\n' \
| tq -p yaml -o json -c '.users[0]'Check truncation before piping a partial document onward:
if ! printf 'items[2]:\n - one\n' | tq check -p toon; then
echo 'truncated input'
fiUpdate in place — tq upgrade resolves the latest release, verifies the download against the release checksums, and replaces its own binary. --check only reports, exiting non-zero when an update is waiting, so it fits in a script:
tq upgrade # no-op when already current
tq upgrade --check # exit 0 up to date, exit 1 update available
tq upgrade X.Y.Z # pin a versionIt honours the same knobs as the installer: TQ_CHANNEL (stable/next), TQ_VERSION (a pin, which the positional argument overrides), and GITHUB_TOKEN. Upgrading needs write permission on the directory holding the binary; without it, tq says so and points at the installer. On Windows a running tq.exe cannot be overwritten, so upgrade renames it aside before writing the new one and cleans the leftover up on the next run.
Source install:
cargo install reddb-io-tqDetails: crates/tq, release assets, TOON format detail, RedDB TOON extensions, TOONL streaming format, and development commands.
Declarative syntax highlighting for .toon and .toonl files, plus toon/toonl fenced code blocks in Markdown. The TextMate grammars cover TOON v4.1 with the RedDB wire extensions, and TOONL v0.1/v0.2 including trailers, continuation headers, named schemas, and tagged rows. Escape mistakes and the reserved TOONL - prefix show up as errors while you type.
Use it when reading or writing TOON documents, TOONL streams, or the spec documents in docs/ inside VS Code.
Every stable release includes the .vsix as a release asset:
curl -fsSL https://github.com/reddb-io/toon/releases/latest/download/reddb-toon.vsix -o /tmp/reddb-toon.vsix && code --install-extension /tmp/reddb-toon.vsixOne-liner from a clone:
(cd packages/vscode-toon && pnpm dlx @vscode/vsce package -o reddb-toon.vsix) && code --install-extension packages/vscode-toon/reddb-toon.vsixVSCodium and Cursor users: swap code for codium / cursor. Once the extension is listed on the Marketplace and Open VSX (planned), the in-editor one-liner becomes Ctrl+P → ext install reddb-io.reddb-toon.
Or open packages/vscode-toon in VS Code and press F5 to try the grammars in an Extension Development Host against examples/sample.toon and examples/sample.toonl.
Details: packages/vscode-toon, TOON spec companion, RedDB TOON extensions, and TOONL streaming format.
Each release publishes tq binaries for Linux, macOS, and Windows, plus checksums and build provenance. The installer script resolves the matching asset for the current platform and installs or updates tq in place.
curl -fsSL https://raw.githubusercontent.com/reddb-io/toon/main/install.sh | shAn installed tq updates itself with tq upgrade, which resolves and verifies the same assets.
Useful installer knobs:
| Variable | Effect |
|---|---|
TQ_VERSION |
Pin a release tag |
TQ_CHANNEL |
Use stable or next |
TQ_INSTALL_DIR |
Choose the installation directory |
TQ_FORCE |
Reinstall even when already current |
git clone https://github.com/reddb-io/toon
cd toon
git submodule update --init
cargo test --workspace
cargo run -p reddb-io-tq -- . deploys.toon
corepack enable
pnpm install
pnpm -r testThe Rust and pnpm workspaces include the format, CLI, RPC, MCP, ACP, and editor packages shown above. Release automation keeps every published crate, npm package, and the extension on the same version.
Auto release is the controller for pushes to main: it derives the stable
SemVer bump from conventional commits, synchronizes every manifest, creates the
release commit, and dispatches the exact SHA. Release is the executor: it
builds platform assets, publishes registries, creates the GitHub Release, and
verifies clean consumers. It can also be dispatched manually for a next
prerelease or to retry a stable release; normal pushes do not run both workflows
independently.
MIT.