Repository navigation
Triage text: failure modes, worker-side regressions, public cloud, virtualization and container guidance - #49
Merged
Merged
Conversation
job-triage.md sat a few bytes under its file cap, so no triage rule could be added to it. "Needle mismatch", "Command failures" and "Log signatures" say what a failing step or log line means once the decision tree has routed it, a topic of its own; they move unchanged to a new reference, and the decision tree, the triage order and the triage playbook point there.
…g else changed "Only BUILD changed: the product changed" gave product verdicts for failures that came from the worker side: a new os-autoinst or QEMU on the workers, the worker that built a published parent disk, or an update server, registry or cloud API that changes outside the build. The rule now asks for the same worker software, read from the os-autoinst line and the worker package diff (for the parent from its log), and for no changed external service, before it names the product. A regression of os-autoinst or other worker software was classed as a test issue for lack of a place; it is INFRA (tools), in the decision tree's classes and in the routing table.
…-markers Three console failures had no row. "typing command '...' timed out" was read as the command failing, although os-autoinst raises it when the echo of the typed line does not come back as typed, before it presses Enter. Keys lost before the target was ready, or a terminal's answerback typed into the shell, looked like product errors. A command that visibly ran without its done-marker was blamed on the product when the test framework had changed. A new "Typed input" section says what each means, the decision tree and the triage playbook route there, and the marker case points at the os-autoinst version compare.
… checks Several recurring step texts had no row, so triages guessed their class: - a library helper's own die after its retries hides the command that failed; read its last attempt and classify that; - a 401 "Unknown Registration Code" is a rotated or invalid code in the settings, which a restart keeps, not an outage; - Secure Boot refusing an unsigned test module is a test issue; - zypper 104 for a package a new major version dropped on purpose is a test issue; - journal_check's "Unknown issue" steps need the ignore list it reads and the test's own actions lined up by time; - a bare-metal host whose GRUB never reaches the SOL console needs the SOL capture and a host comparison. The "timed out" bullet now points at the stall rule as well, which names an overloaded worker host as the usual cause of a stall.
…pe runs "Machine didn't shut down!" was routed to the needle rules, which do not apply to a shutdown that a unit or mount held up; it gets a row of its own that names the screen, serial and journal. A user_cancelled job had no route at all, although it is often cancelled after a hang whose first error is the finding. And no check asked whether the scenario should run on that product or version at all, so failures whose fix was a schedule change were classified as product or test code.
"Incomplete reasons" called a failed asset download a TEST settings issue while the routing table in review-workflow.md called it infrastructure, so triages of the same failure disagreed. Both are right in turn: a new build's asset that was never published or synced is infrastructure, a setting that names the wrong asset is a test issue. The row now names the check that decides, whether the asset exists for that build, and the routing table points to it.
The decisive evidence of several failures sat in uploads the artifact map did not name: the SUT journal a failing module uploads, the SOL capture of a bare-metal host whose boot failed before SSH came up, and the public-cloud uploads, where img-proof's own output, the SUT's journal inside the supportconfig archive and the instance console are, while the parsed step only says "fail". Each gets a row that says when to open it.
Three readings of openqa-investigate's clones misled triages. The last-good-build clones copy the last good job, its settings with it, so a settings change looked like a product change. A passing clone was taken as proof for a failure that hits most runs but not all, where one pass is chance. And a triaged job that was itself an investigation clone was given its own window, which it has none of; oqa-job.py names its origin, whose scenario holds the history.
…eout A review of an older failure followed SUPERSEDED to a restart made after the cutoff and so judged the failure by later evidence, although oqa-job, oqa-history and oqa-sweep now take --before. The triage order says when to use it and the playbook shows it, with --max-line-chars 0 for lines the default cuts. After a timeout the decisive delay can sit in an earlier module that passed, which the failing step never shows, so the module runtimes against last good come first.
The triage fallback "without the scripts" read as leave to call the raw API once a script call had been denied, which the untrusted-content policy forbids: a denial is final. The fallback is now for missing scripts only, in job-triage.md, the API recipes and the playbook, and a denial is reported instead. Breadth checks fetched raw /api/v1/jobs lists, up to tens of MB, which carry every job setting with registration codes, and saved them. Breadth now goes through oqa-sweep.py with --test and --before, and the references say not to fetch or save such lists.
area-conventions.md had no room left under its file cap for the triage parts of the public cloud, virtualization and container areas. Unit tests are a contribution rule for several areas with their own local gate, which is what contributing-gates.md owns, so the section moves there unchanged and the two pointers to it follow.
Public-cloud failures were triaged from steps that say only "fail", while the cause sat in the uploads or in the cloud around the job. The section now says where img-proof's real output is, how a registration failure of the client or image differs from an outage of the cloud's update servers, which provisioning errors are a capacity shortage the zone and region fallbacks could not avoid, what an nc "timed out" means against "refused", that SLE Micro up to 5.4 has no cloud-init, and that the image under test is named by the PUBLIC_CLOUD_* image settings rather than by BUILD.
A suite or kselftest that stops compiling with one kernel-source build and not with its neighbours points to a backport missing in that build, which triages read as a suite or test problem. The kernel triage order now says to compare the same build step across neighbouring builds.
… are The only line pointing at a qemu crash on the host was "post_fail_hook failed: COREDUMPS found", which the triage references called secondary, like any death of the hook. A core dump the hook reports names the process that crashed and can be the cause. "Virtualization" now says where the host's libvirt, qemu and journal uploads and the core dump reports are, and the log signature and pitfall for hook failures make that exception.
bci_version_check fails when the moving totest tag already points to a newer build than the one the job was scheduled for, or times out pulling while that build is prepared. Triages took it for a product bug. "Containers" now says it is a race that makes the job obsolete, and the decision tree routes its "Stale build" step there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Triaging failures across QE areas showed rules that pointed the wrong way or were missing: "only BUILD changed" ignored worker software and external services, test-engine regressions had no class, and console typing, journal-check, Secure Boot, shutdown, cancellation and asset cases had no row, while public-cloud, virtualization and container triage had no guidance. This moves the console and command rows into a new references/failure-modes.md and adds those rules, points to the new
--beforeviews for as-of reviews, and makes a denied script call final instead of a reason to fetch raw job lists.