gk7205v200: capture a crash log across reboot with pstore/ramoops - #2392
Conversation
A camera that boots but whose kernel panics or oopses (a driver/sensor init
loop, a crashing streamer) leaves nothing behind: the console is muted, syslog
is a RAM ring, and the panic=20 reboot wipes it. Enable pstore/ramoops so the
last kernel log survives the warm reboot and is readable at /sys/fs/pstore on
the next boot -- via kmsg_dump, which is independent of the muted console, so it
works on these boards. (A cold power cut still loses it; that case is the
bootloader firmware_scan recovery, not this.)
128 KiB is enough for one 64 KiB panic dump plus headroom (record_size 64K,
mem_size 128K, console/pmsg zones off). The region is carved so it collides
with neither Linux RAM nor the media (mmz) pool, in both goke allocators:
gk (mem=osmem, mmz above): shrink the OS window by 128K and place ramoops in
the freed gap [base+osmem-128K, base+osmem), just below mmz_start, which
is unchanged -- the mmz pool is untouched.
cma (mem=totalmem, mmz a CMA region inside it): reserve the top 1 MiB (mmz is
M-granular there; a >=128 MiB cma board loses 1 MiB, negligible), drop
mem= by the same 1 MiB, and place ramoops at the top, above both the
kernel window and the media zone.
set_allocator writes the ramoops.* params (and the reduced mem=) into bootargs;
load_goke's check_allocator backfills them on a camera whose bootargs predate
pstore (set_allocator otherwise only runs when mmz_allocator is absent), so the
region reaches already-provisioned cameras on their next boot too. Effective on
the boot after the rewrite, like every other set_allocator change.
Scoped to gk7205v200 (its own osdrv package). Shell-parse and strip-comment
gates pass; address math verified for both allocators. On-camera panic ->
pstore verification under each allocator follows.
PR Summary by QodoPersist gk7205v200 kernel crash logs with pstore/ramoops
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
…targs (review)
Two fixes from review of the pstore change:
- Nothing mounted the pstore filesystem, so a crash log ramoops captured across
the reboot was held in the reserved RAM but never exposed at /sys/fs/pstore.
Add S29pstore, which mounts it when the kernel has pstore (inert otherwise),
the same shape as S29debugfs. (finding: crash logs stay hidden after reboot)
- check_allocator backfilled the ramoops bootargs on already-provisioned
cameras by re-running set_allocator, but set_allocator rebuilds bootargs from
a fixed set of parsed fields plus ${extras}, so any other custom kernel
parameter stored directly in bootargs would be dropped on the next boot.
Revert the backfill: ramoops is now added only through set_allocator's
existing trigger (when mmz_allocator is absent -- a fresh env / firstboot), so
a provisioned camera's bootargs are never rewritten. Fleet reach for existing
cameras is deferred rather than bought at the cost of clobbering their kernel
command line. (finding: upgrades erase custom boot options)
The disable line targeted CONFIG_PSTORE_DEFLATE_COMPRESS, a symbol this 4.9 kernel does not have (it was renamed from ZLIB in a later kernel), so it was a no-op. The compressor here is a mandatory Kconfig choice with no "none" arm; it defaults to ZLIB, whose workspace is preallocated at pstore_register so it is safe to compress from inside a panic. Name that symbol explicitly instead of a dead one. Resolved kernel config is unchanged (ZLIB was already selected).
Hardware verification (lab gk7205v200) — the bootargs carve bricks the camera; converting to draftRan the memory-map test the review asked for, on the lab gk7205v200 (UART + PoE recovery). Result: fail on the What happens. After Isolation (what is and isn't the cause):
Conclusion. The What is verified-good and can be kept (all confirmed on hardware): the Marking this draft while the carve is redesigned. The lab camera was recovered to a healthy |
The flash-root test grepped the whole kernel command line for a bare "ram"/"mmcblk"/"nfs", so any unrelated token spelled with those letters made init believe the root was a ramdisk / SD / NFS mount. It then skipped the flash overlay mount and pivot_root entirely and fell through to /sbin/init on the read-only squashfs: the camera came up with a read-only /etc, its saved password gone (unclaimed), and -- because /etc/fw_env could not be written -- load_goke fell back to the default osmem and the mmz zone collided with kernel RAM, so nothing streamed. The trigger was the pstore "ramoops.*" crash-log parameters, but the bug is the unanchored match; anchor all three to the root= value. A flash board's cmdline carries none of these letters in root=, so its result is unchanged; only a command line that mentioned them outside root= (as ramoops does) is fixed. Found on a gk7205v200 (UART recovery) while bringing up pstore.
…rounding and U-Boot set_allocator cma hung the board right after "Starting kernel" -- an early panic() before the console. Two constraints, both found on hardware, decide where ramoops can go on the cma path: - The media zone is a cma region, reserved with dma_declare_contiguous(), which rounds the region up to the CMA alignment (PAGE_SIZE << max(MAX_ORDER-1, pageblock_order) = 4M on this kernel) and rejects a fixed region whose rounded end runs past the end of RAM (mm/cma.c: "fixed && base + size > highmem_start"). drivers/goke/cma/cma.c turns that -EINVAL into a panic() from an early initcall. The first attempt shrank the zone by 1M, which rounded back up to the full size and overran mem= by 1M -> panic. The zone has to end on a 4M boundary at or below mem=, so drop a whole 4M (totalmem-osmem-4, still 4M-aligned since osmem/totalmem are 4M multiples). - ramoops then cannot go in the freed top 128K either: U-Boot relocates to the top of DRAM on every boot (bdinfo here: relocaddr 0x43F20000, page tables 0x43FF0000 -- the top ~1.25M), so a crash log left there is overwritten by the reboot before the kernel reads it back (seen as an empty /sys/fs/pstore after a panic). Land ramoops at the media zone's end instead and stop mem= there, well clear of both the kernel's RAM and U-Boot's window. On a cma board (>=128M) the 4M is negligible; gk7205v200 is 64M and always runs the gk allocator, so this branch is only reached by a manual override. Verified on a gk7205v200: cma now boots, streams, and captures a panic to /sys/fs/pstore across the reboot.
Resolved — root-caused and fixed in-repo; both allocators verified on hardwareMy earlier "the carve bricks the camera" comment was premature. On the lab gk7205v200 (UART + PoE) I found the two real root causes and fixed them; pstore now works end-to-end on both allocators, no brick. Root cause 1 — the gk brick was a shared-init bug, not the carve
Root cause 2 — cma hung at "Starting kernel" from CMA alignment, not ramoopsThe cma media zone is a CMA region reserved via Hardware evidence (lab gk7205v200)gk (the board's real allocator): cma (manual override): Also confirmed on hardware: |
|
Code review by qodo was updated up to the latest commit 956ddb3 |
…evice tree The ramoops region is now reserved by a /reserved-memory node in the board DTS (OpenIPC/linux), which costs exactly 128K and leaves mem= and the media (mmz/cma) zone untouched, so it works for both allocators. Carving it out of bootargs here could not: on the cma path the media zone is a CMA region that dma_declare_contiguous() rounds up to the 4M CMA alignment, so making room for a 128K log cost a full 4M and, done wrong, paniced the board before the console. Revert set_allocator to exactly its pre-pstore form (no ramoops parameters, no mem/media reduction). CONFIG_PSTORE_RAM and the S29pstore mount stay; the driver binds to the DT region. Needs the OpenIPC/linux DTS node to actually capture a crash, but is harmless without it (pstore just mounts empty). No conflict in either landing order, since bootargs no longer carry ramoops.* to clash with the DT node.
Pivot: ramoops region now comes from the device tree, not a bootargs carvePer review discussion, the ramoops reservation is moving to a This PR is now reduced to its safe, in-repo pieces (
Verified on hardware (gk7205v200) with the DTS node + plain bootargs — full
Coordination: this and OpenIPC/linux#56 can land in either order — the firmware no longer emits |
1 participant
Why
A camera that boots but whose kernel panics/oopses (a driver/sensor init loop, a crashing streamer) leaves nothing to diagnose: the console is muted, syslog is a 64 KB RAM ring, and the
panic=20reboot wipes both. Enable pstore/ramoops so the last kernel log survives the warm reboot and is readable at/sys/fs/pstorenext boot — captured viakmsg_dump, which is independent of the muted console. (A cold power cut still loses it; that case is the bootloader recovery, not this.)The "collect logs to root-cause a field brick" half of the initiative.
What
Three files. The ramoops region is reserved in the device tree (OpenIPC/linux), not carved from bootargs:
gk7205v200.generic.config— enableCONFIG_PSTORE+PSTORE_RAM(+ PMSG; ZLIB compressor, the only one whose workspace is preallocated at register time, so it is safe to compress from inside a panic on this 4.9 kernel).general/overlay/etc/init.d/S29pstore— mount/sys/fs/pstoreso the records are exposed (review finding 2).general/overlay/init— anchor the flash-vs-ramdisk match to theroot=value: a bareramwas matching the substring inramoops, so init skipped the overlay pivot and fell through to the read-only squashfs. A correctness fix for every board.The physical reservation is a
/reserved-memoryramoops node in OpenIPC/linux (merged intogoke-gk7205v200): it costs exactly its own size, leavesmem=and the mmz/cma media zone untouched, and works for both goke allocators without the CMA-alignment loss of a bootargs carve. This supersedes the earlier bootargs-carve approach, which bricked the gk allocator — see the thread below for the full root-cause (the carve is gone;set_allocatoris back to its original form).Hardware verification (lab gk7205v200, both allocators)
set_allocator gk→ claimed,pstore mounted, MMZ conflict 0, RTSP up.echo c > /proc/sysrq-trigger→ after the reboot/sys/fs/pstore/dmesg-ramoops-0(13 KB) holds the crash (sysrq_handle_crash+ backtrace); streaming intact.