gk7205v200: failsafe rescue mode + flash-free DRAM boot counter + crash log

Builds the recovery loop on top of the pstore region (#2392) and the bootcount
arm/disarm (#2391), converting the counter off the NOR env.

- Move the boot counter from the U-Boot env to a single DRAM word at 0x41f20000
  (a no-map reserved region, board DTS): S99bootok clears it through /dev/mem
  once majestic is proven healthy, and sysupgrade arms it the same way. Writing
  the env every boot is NOR wear and a mid-write brick risk; a healthy boot now
  writes zero flash. On a U-Boot/kernel without the region the write lands in
  spare reserved RAM -- harmless.
- rcS gains a failsafe branch: when U-Boot appends "failsafe" to the cmdline
  after bootlimit failed boots, bring up only logging, mdev, networking and SSH
  -- skip the SDK and the streamer -- so a crashlooping camera lands reachable
  instead of looping. The overlay stays mounted (claim intact); the counter is
  reset on entry so failsafe does not re-escalate.
- S98crashlog preserves a crashed boot's pstore log into /etc/crash on the next
  normal boot (gzipped, capped to the latest, pstore ring then freed) and rcS
  drops a breadcrumb when it fell into failsafe, for the WebUI to offer the
  owner. Overlay-only, no flash on a clean boot.

Proven on hardware, both allocators: gk7205v200 (64M) and gk7205v300 (128M) --
crashloop escalates to failsafe and self-recovers, pstore panic is harvested,
counter resets on a healthy boot.

Author: widgetii

br-ext-chip-goke/board/gk7205v200/gk7205v300.generic.config

@@ -2393,7 +2393,16 @@ CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=1
 # CONFIG_QNX4FS_FS is not set
 # CONFIG_QNX6FS_FS is not set
 # CONFIG_ROMFS_FS is not set
-# CONFIG_PSTORE is not set
+CONFIG_PSTORE=y
+# Compression is a mandatory choice on this 4.9 kernel (it has no "none"); ZLIB
+# is its default and the only one whose workspace is preallocated at register
+# time, so it stays safe to compress from inside a panic. The others stay off.
+CONFIG_PSTORE_ZLIB_COMPRESS=y
+# CONFIG_PSTORE_LZO_COMPRESS is not set
+# CONFIG_PSTORE_LZ4_COMPRESS is not set
+CONFIG_PSTORE_CONSOLE=y
+CONFIG_PSTORE_PMSG=y
+CONFIG_PSTORE_RAM=y
 # CONFIG_SYSV_FS is not set
 # CONFIG_UFS_FS is not set
 CONFIG_NETWORK_FILESYSTEMS=y

general/overlay/etc/init.d/S98crashlog

@@ -0,0 +1,37 @@
+#!/bin/sh
+# After a crash, preserve the pstore log where the WebUI can offer it to the
+# owner, then free the pstore ring for the next one. pstore itself survives a
+# warm reset but not a cold power-cycle; copying the record into the overlay
+# lets the notice and the download outlive the plug being pulled. The bundle is
+# capped to the latest crash (overwritten each time), so the overlay never grows
+# without bound, and a clean boot -- no records -- writes nothing at all. This is
+# the capture half of the owner-pull crash report (WebUI is the other half); it
+# pairs with the failsafe breadcrumb in rcS and with S99bootok.
+case "$1" in
+	start|"") ;;
+	*) exit 0 ;;
+esac
+
+PSTORE=/sys/fs/pstore
+CRASH=/etc/crash
+
+# Nothing captured last boot -> ordinary boot, leave no trace.
+ls "$PSTORE"/dmesg-* >/dev/null 2>&1 || exit 0
+mkdir -p "$CRASH" 2>/dev/null || exit 0
+
+# busybox tar has no built-in -z here, so compress through gzip and confirm the
+# result is non-empty before trusting it.
+if tar cf - -C "$PSTORE" . 2>/dev/null | gzip > "$CRASH/crash.tar.gz.tmp" 2>/dev/null \
+	&& [ -s "$CRASH/crash.tar.gz.tmp" ]; then
+	mv "$CRASH/crash.tar.gz.tmp" "$CRASH/crash.tar.gz" 2>/dev/null
+	{
+		echo "utc=$(date -u '+%Y-%m-%d %H:%M:%S' 2>/dev/null)"
+		echo "records=$(ls -1 "$PSTORE"/dmesg-* 2>/dev/null | wc -l)"
+		echo "bytes=$(wc -c < "$CRASH/crash.tar.gz" 2>/dev/null)"
+	} > "$CRASH/pending" 2>/dev/null
+	logger -s -t crashlog "preserved pstore crash log to $CRASH/crash.tar.gz" 2>/dev/null
+	rm -f "$PSTORE"/dmesg-* 2>/dev/null
+else
+	rm -f "$CRASH/crash.tar.gz.tmp" 2>/dev/null
+fi
+exit 0

general/overlay/etc/init.d/S99bootok

@@ -1,27 +1,30 @@
 #!/bin/sh
-# Disarm the bootloader's boot-count recovery once the camera has reached a
-# healthy userspace. sysupgrade sets upgrade_available=1 before a flash; a
-# U-Boot that supports boot-count recovery keeps counting boots until this
-# clears the flag, and reflashes a known-good image if a broken one never gets
-# here. On a U-Boot without boot-count logic the variable is ignored, so this
-# is a no-op beyond one environment read. A normal (unarmed) boot costs a
-# single fw_printenv and writes nothing.
+# Clear the bootloader's boot counter once the camera has reached a healthy
+# userspace. U-Boot increments a counter every boot and, if it is never cleared,
+# boots FAILSAFE (network + SSH only) after bootlimit tries -- so a camera that
+# boots but never streams lands somewhere reachable instead of looping forever.
+#
+# The counter is a single DRAM word (a no-map reserved region at 0x41f20000, see
+# the board DTS), NOT the U-Boot env: we clear it through /dev/mem, writing no
+# flash at all. On a U-Boot/kernel without this the write just lands in spare
+# reserved RAM nobody reads, so it is harmless. A normal boot writes nothing to
+# flash, ever.
 case "$1" in
 	start|"") ;;
 	*) exit 0 ;;
 esac
 
-command -v fw_printenv >/dev/null 2>&1 || exit 0
-[ "$(fw_printenv -n upgrade_available 2>/dev/null)" = "1" ] || exit 0
+command -v devmem >/dev/null 2>&1 || exit 0
+BOOTCOUNT_ADDR=0x41f20000
 
-# Armed. Disarm only once the new image is PROVEN healthy -- and prove it,
-# don't just glimpse it: pidof sees majestic the instant S95majestic forks,
-# before the pipeline is up, so a streamer that crashes during startup would
-# look healthy for a moment. Require it to stay up across a sustained window;
-# if it dies the window restarts, and if it never holds we leave the flag armed
-# so U-Boot escalates to recovery instead of looping on an image that boots but
-# does not stream. An image built without majestic is healthy once it boots
-# this far. Backgrounded so nothing here holds up boot.
+# Clear only once the image is PROVEN healthy -- and prove it, don't just glimpse
+# it: pidof sees majestic the instant S95majestic forks, before the pipeline is
+# up, so a streamer that crashes during startup would look healthy for a moment.
+# Require it to stay up across a sustained window; if it dies the window
+# restarts, and if it never holds we leave the counter so U-Boot escalates to
+# failsafe instead of looping on an image that boots but does not stream. An
+# image built without majestic is healthy once it boots this far. Backgrounded
+# so nothing here holds up boot.
 {
 	need=8       # consecutive good checks (~16s) before trusting the image
 	ok=0
@@ -30,13 +33,7 @@ command -v fw_printenv >/dev/null 2>&1 || exit 0
 		if [ ! -x /usr/bin/majestic ] || pidof majestic >/dev/null 2>&1; then
 			ok=$((ok + 1))
 			if [ "$ok" -ge "$need" ]; then
-				if fw_setenv upgrade_available 0 2>/dev/null &&
-					fw_setenv bootcount 0 2>/dev/null; then
-					exit 0
-				fi
-				# Environment readable but not writable: say so, or every
-				# healthy boot silently leaves U-Boot counting toward recovery.
-				logger -s -t bootok "healthy but could not clear upgrade_available (U-Boot env not writable) -- bootloader will keep counting" 2>/dev/null
+				devmem "$BOOTCOUNT_ADDR" 32 0 2>/dev/null
 				exit 0
 			fi
 		else
@@ -45,5 +42,5 @@ command -v fw_printenv >/dev/null 2>&1 || exit 0
 		sleep 2
 		tries=$((tries + 1))
 	done
-	logger -s -t bootok "majestic did not stay up; leaving boot-count recovery armed" 2>/dev/null
+	logger -s -t bootok "majestic did not stay up; leaving the boot counter for U-Boot to escalate to failsafe" 2>/dev/null
 } &

general/overlay/etc/init.d/rcS

@@ -5,6 +5,36 @@ export UPGRADE=$(fw_printenv -n upgrade)
 # that an unset TZ would use, so a missing /etc/TZ must leave TZ alone.
 [ -r /etc/TZ ] && export TZ=$(cat /etc/TZ)
 
+# Failsafe rescue mode. The bootloader appends "failsafe" to the kernel command
+# line after repeated failed boots (and it can be set by hand); here we bring up
+# only logging, device nodes, networking and SSH, and skip the SDK (S70vendor),
+# the streamer (S95majestic) and every other service, so nothing that could be
+# crashlooping runs. The overlay is still mounted (see /init), so the camera
+# stays claimed -- SSH auth works -- and its config and /sys/fs/pstore logs are
+# left intact for inspection. Nothing is wiped: a reachable camera to diagnose
+# and fix, instead of an unreachable one someone has to reset "blind".
+if grep -qw failsafe /proc/cmdline 2>/dev/null; then
+	echo "*** FAILSAFE: bringing up network + SSH only, services skipped ***"
+	for s in S01syslogd S29pstore S38mdev S31hostname S40network S50dropbear; do
+		[ -x "/etc/init.d/$s" ] && "/etc/init.d/$s" start
+	done
+	# Leave a breadcrumb the WebUI reads on the next normal boot, so a fault that
+	# left no pstore panic -- a service that simply never came up -- still shows
+	# the owner "this camera recovered from a crash". Overlay-only, no flash.
+	mkdir -p /etc/crash 2>/dev/null && \
+		printf 'reason=bootlimit\nutc=%s\n' \
+			"$(date -u '+%Y-%m-%d %H:%M:%S' 2>/dev/null)" \
+			> /etc/crash/failsafe 2>/dev/null
+
+	# Reaching failsafe counts as "handled": clear the bootloader's DRAM boot
+	# counter (no flash) so once the operator fixes the fault and reboots, the
+	# next boot is a normal one. If they reboot without fixing it, three more
+	# failed boots simply land back here. Cold power-cycle also clears it.
+	command -v devmem >/dev/null 2>&1 && devmem 0x41f20000 32 0 2>/dev/null
+	echo "*** FAILSAFE: ready. SSH in to inspect (pstore, config) and recover. ***"
+	exit 0
+fi
+
 for i in /etc/init.d/S??*; do
 	[ ! -f "$i" ] && continue
 	case "$i" in

general/overlay/usr/sbin/sysupgrade

@@ -98,13 +98,9 @@ die() {
 	if [ "1" = "$flash_touched" ] || [ "1" = "$_ramfs_phase" ]; then
 		reboot_system
 	else
-		# Nothing was written, so the camera is unchanged: drop any recovery arm
-		# set before the flash phase, or a boot that never flashed would still be
-		# counted. (A failure AFTER enter_ramfs reboots via the branch above
-		# instead, and S99bootok disarms on the next healthy boot -- bootcount
-		# only advances one per reboot and the limit is several, so no false
-		# escalation.)
-		command -v fw_setenv >/dev/null 2>&1 && fw_setenv upgrade_available 0 2>/dev/null
+		# Nothing was written, so the camera is unchanged. The DRAM boot counter
+		# is cleared only on the flash path below (and clearing it is harmless
+		# even if no flash follows), so there is nothing to undo here.
 		restore_resources
 		# Only ever remove a lock this process owns. die() is reachable before
 		# create_lock has run (get_system_info, called from print_sysinfo, can
@@ -1348,23 +1344,21 @@ if [ "1" = "$skip_reboot" ] && [ "1" = "$root_on_flash" ] &&
 	sleep "$abort_wait"
 fi
 
-# Arm the bootloader's boot-count recovery for this flash. If the freshly
-# written image never reaches a healthy userspace, a U-Boot that supports
-# boot-count recovery falls back to reflashing a known-good image instead of
-# rebooting into the broken one for ever; S99bootok disarms this once majestic
-# is up. On an older or vendor-supplied U-Boot with no boot-count logic these
-# are unused environment variables it ignores, so arming is harmless there.
-# Set here, on the still-normal system, before enter_ramfs moves the MTD
-# environment out of reach.
+# Clear the bootloader's boot counter so the freshly written image starts with a
+# clean slate: if it never reaches a healthy userspace, U-Boot boots FAILSAFE
+# (network + SSH only) after bootlimit tries instead of looping on a broken
+# image, and S99bootok clears the counter again once majestic is up. The counter
+# is a no-map DRAM word (0x41f20000, see the board DTS) -- this writes no flash,
+# unlike an env-backed counter whose per-boot writes wear NOR and risk a bricked
+# env on a power cut. On a U-Boot/kernel without it the write lands in spare
+# reserved RAM nobody reads. Done on the still-normal system, before enter_ramfs
+# moves things out of reach.
+command -v devmem >/dev/null 2>&1 && devmem 0x41f20000 32 0 2>/dev/null
+
+# A camera that ever ran the pre-verify U-Boot has verify=n saved in flash,
+# where it still overrides the built-in default; drop the key so the kernel
+# image is CRC-checked on boot again.
 if command -v fw_setenv >/dev/null 2>&1; then
-	if fw_setenv upgrade_available 1 2>/dev/null && fw_setenv bootcount 0 2>/dev/null; then
-		:
-	else
-		echo_c 33 "Warning: could not arm boot-count recovery (U-Boot env not writable?); the upgrade proceeds without it."
-	fi
-	# A camera that ever ran the pre-verify U-Boot has verify=n saved in flash,
-	# where it still overrides the built-in default; drop the key so the kernel
-	# image is CRC-checked on boot again.
 	fw_setenv verify 2>/dev/null ||
 		echo_c 33 "Warning: could not clear a saved verify=n; a corrupt kernel may not be caught on boot."
 fi