Decode PDF text strings in metadata and attachment names

[unidoc-anom]

Description and links

PDF text strings are commonly UTF-16BE with a leading BOM, or PDFDocEncoded. We were returning several of them as raw bytes, so a non-ASCII document title or attachment name came out as mojibake in both the CLI and the GUI. A German or CJK /Info /Title showed up as FEFF0052006500630068... in dump metadata --json, which is indistinguishable from a document that genuinely has that string as its title.

This routes the human-text fields through one shared decoder so a string field in the machine contract is actually the string.

Technical changes

1. Moved decodeTextString out of validate.go into a new internal/pdfcore/textstring.go, and added decodeTextStringOK which returns an explicit success bit alongside the value. checkXMPMetadata still calls the same symbol with no behavior change.
2. Added a textStringOrRaw wrapper for display fields. It falls back to the raw rendering when a decode fails, so an undecodable field degrades to visible mojibake instead of disappearing from the output.
3. Wired the wrapper into collectInfoFields for the six /Info text keys (Title, Author, Subject, Keywords, Creator, Producer) and into embeddedFileFromFilespec for the filespec /UF and /F display names.
4. Added asciiSafe to cmd/cli/output.go and applied it to the plain-text Info values and the embedded-file table's Name, Relationship and MIME cells. It only escapes when a value carries a byte outside 0x20-0x7e, so existing all-ASCII output is byte-identical.
5. dump embedded --name with no match now prints a hint pointing at --json when the document has names the table can't show verbatim.

Considerations

The fallback encoding is Latin-1, not CP1252, despite pdfcpu's function being named CP1252ToUTF8 — it's utf8.EncodeRune(rune(s[i])) per byte, an identity map with a misleading name. StringLiteral("\x80") decodes to U+0080, not the CP1252 euro sign. I went back and forth on this one and only settled it by running the decoder. Accepting pdfcpu's behavior as-is rather than hand-rolling an encoding table; the unit table pins what it actually does so nobody "fixes" it later.

Two things deliberately left alone:

• tree.go scalarDisplay renders every string in the object tree and has no key context, so it can't tell a text /Title from a binary /ID. Routing it through a text decoder would corrupt binary displays. Decoding there needs key-aware plumbing and is a separate change.
• Binary-carrying fields in the same dicts stay raw: /Params /CheckSum, signature /Contents and /Cert, the trailer /ID.

The decoder now guarantees valid UTF-8 on both branches. json.Marshal silently substitutes one U+FFFD per invalid byte, which destroys the value on the surface that's supposed to be machine-readable — worse than the hex rendering it replaced, because hex is at least recoverable. A decode that succeeds but yields invalid UTF-8 counts as a failure, and a raw fallback that isn't valid UTF-8 gets hex-encoded. Worth noting the second case predates this change; I fixed it because leaving one branch of a new guard unvalidated didn't make sense.

Also filed follow-ups for a few adjacent things I found and didn't fix here: PDF date strings are legally text strings so a <FEFF...> /ModDate still renders as hex; tableWriter has no cell cap so an escaped CJK name pushes later columns off screen; and cmd_font.go's roster table has the same unescaped-cell issue this fixes for dump embedded.

How was this tested?

• go vet ./... clean, golangci-lint run reports 0 issues
• go test -race ./... — 7/7 packages
• Per-suite loop over tests/*/ — 49/49
• npx tsc --noEmit clean, npm test 833/833 in 59 files, eslint . --max-warnings 0 clean
• New coverage: 19 unit tests in internal/pdfcore/textstring_test.go, 9 CLI integration tests in tests/shared-text-string-decoder/ against the built binary, 4 presenter tests in cmd/cli/output_test.go
• New fixture testdata/correctness/text-string-encoding.pdf with a UTF-16BE /Info /Title and a non-ASCII filespec /UF, documented byte-by-byte in the README
• Manual: dump metadata and dump embedded against the fixture, both with and without --json, plus dump object --ref and dump tree to confirm the raw renderers are unchanged

I mutation-tested the production surface rather than trusting the tests by inspection — 32 mechanical mutations across the seven touched files, 30 caught. The two survivors led to real gaps: the /Params /CheckSum and /ModDate guards couldn't fail if you wired those fields through the decoder, because their fixture values are printable ASCII or hex-fall-back to the same digits. Same story for the /F decode branch, which every existing fixture skipped by carrying a /UF. Those are pinned now, and I verified each new guard fails when you revert the behavior it guards.

What could go wrong?

Two user-visible CLI changes that could break a script:

• dump embedded --name now matches the decoded name. A caller passing the old raw hex text won't match anymore.
• A non-ASCII value on the plain-text surface is now wrapped in strconv.QuoteToASCII form. All-ASCII values are untouched, and plain text is documented as non-contractual, but anything parsing it should be on --json anyway.

The GUI views render the decoded values through the same structs, but there's no IPC or binding change so nothing here exercises them — worth opening a PDF with a non-ASCII title in the app before release.

dump metadata's XMP packet is still passed through verbatim and is UTF-8 XML by spec, so that command's plain output isn't strictly ASCII-only on a document that has an XMP packet. Escaping it would corrupt it as XML. That's documented in the presenter and the assertion is scoped to the Info block rather than claiming something the code doesn't do.

Screenshots/videos (if appropriate)

n/a — CLI and data-layer change.

Checklist

• Tests pass locally
• I targeted the dev branch, not master

[Copilot]

Pull request overview

This PR fixes mojibake for human-readable PDF text strings by routing selected metadata and embedded-filespec display fields through a shared PDF text-string decoder, while keeping binary-carrying fields and raw renderers unchanged. It also makes the CLI plain-text surfaces ASCII-safe without changing existing all-ASCII output.

Changes:

• Introduces internal/pdfcore/textstring.go with decodeTextStringOK and textStringOrRaw, and wires decoding into /Info text keys and filespec /UF and /F.
• Escapes non-ASCII/non-printable values on CLI plain-text surfaces via asciiSafe, including embedded-file table cells.
• Adds unit, integration, and presenter tests plus a documented correctness fixture to pin decode semantics, boundaries, and fallback behavior.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/14-4-shared-text-string-decoder/text_string_decoder_test.go	New black-box CLI integration tests pin decoded JSON and ASCII-only plain output behavior.
tests/14-4-shared-text-string-decoder/helpers_test.go	Harness utilities to build/run the CLI and parse JSON outputs against the fixture.
tests/14-4-shared-text-string-decoder/go.mod	New standalone test module for the acceptance suite.
testdata/correctness/README.md	Documents the new text-string-encoding.pdf fixture and its byte-level intent.
internal/pdfcore/validate.go	Keeps XMP validation using the shared decoder after moving it into a helper file.
internal/pdfcore/textstring.go	New shared PDF text-string decode helper plus safe display fallback (textStringOrRaw).
internal/pdfcore/textstring_test.go	Extensive unit coverage for decoder semantics, call-site wiring, and binary boundaries.
internal/pdfcore/metadata.go	Decodes the six /Info text keys via textStringOrRaw while leaving date keys raw.
internal/pdfcore/embedded.go	Decodes filespec /UF and /F display names while keeping /Params fields raw.
cmd/cli/output.go	Adds asciiSafe to conditionally escape non-ASCII/non-printable bytes on plain output.
cmd/cli/output_test.go	Adds tests pinning asciiSafe behavior and XMP-vs-Info plain output split.
cmd/cli/cmd_metadata.go	Applies asciiSafe to Info block values on the plain dump metadata surface.
cmd/cli/cmd_embedded.go	Escapes embedded table cells and adds a --json hint when name matching may differ.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.

[anovik]

The changes look good but most of the comments are too verbose. Could you please go through the added comments so that they are not too long and don't contain historical information?

[unidoc-anom]

@anovik Done — went through every comment this branch adds.

• Dropped the historical narrative entirely (no more "before X this returned Y", "pre-existing, not introduced by", or notes about what an earlier version did wrong). Where a later change altered behaviour I rewrote the comment to describe the current behaviour instead of appending to it.
• Removed the traceability IDs and priority tags from comments, test names and assertion messages. The helper that only existed to carry an ID into failure messages is gone too — t.Helper() already attributes the failure to the calling test.
• Renamed tests/14-4-shared-text-string-decoder to tests/shared-text-string-decoder to match the unnumbered suites, and cleaned the last references out of the fixture README.
• Trimmed the long godocs. textstring.go went from 84 comment lines to 41, and the unit suite from 273 to ~79; what's left states the behaviour and the couple of caveats a caller actually needs (the two literal types decode differently, the result is always valid UTF-8, indirect objects aren't dereferenced).

[anovik]

LGTM