testdata: the reference chain's frames as gold standards, with a check - #49
hansfbaier wants to merge 6 commits into
Conversation
|
While this PR's CI ran I went through the four failures, and three of them are main's rather than this PR's. The workflow history is the clearest evidence:
So main has been red since 11:56, and the failures in my run are the same ones:
If you want, I will fix those two in a separate commit here — the |
…t wrong
Comparing both chains over the 21 demo-projects designs that carry a .fasm
showed 19 of them assembling to a different configuration and 2 to no bitstream
at all. This keeps the inputs behind that together with the frames
fasm2frames produced for them, and a check that runs fpga-as on the same input
and the same database and compares the configuration bits:
check.sh 5 cases, 5 pass with the fixed assembler and 5 fail with the
assembler built from b234841 (before the change)
cases.tsv part, family and expectation per case
<case>/ input.fasm and, where the reference assembles it, gold.frames
The database is not committed (hundreds of MB), so the check reads
PRJXRAY_DB_DIR and reports skipped without it, which keeps it safe to call from
anywhere. It is a script rather than a bazel test on purpose: Bazel 9 has no
native sh_test and @rules_shell is not in this module graph.
The raw evidence -- the failed bitstreams, the payload byte diffs between the
two builds, the stderr of all three, and the one case that still disagrees with
the reference -- is in
https://gist.github.com/hansfbaier/20a7b7da75d2bf8fc28ad92936d620e2
Assisted-by: deepseek/deepseek-flash
The pull-up templates are runtime strings, and absl::StrFormat wants a format string that is constexpr: newer absl marks that overload unavailable, so injected-features.cc fails to compile under clang 21 (CI's MacOsBuild job) and clang-tidy reports it as an error. absl::Substitute takes the format string at run time by design, so the templates use $0/$1 placeholders and the three call sites call Substitute. Behaviour is unchanged: with and without --emit_pudc_b_pullup the frames are identical to the binary built from main (44 and 46 non-zero frames respectively), and the reference-parity check passes 5/5. Assisted-by: deepseek/deepseek-flash
The clang-tidy --fix commit (957dafc) rewrote std::search(words_.cbegin(), words_.cend(), kWcfgCmd.cbegin(), kWcfgCmd.cend()) into std::ranges::search(words_, kWcfgCmd), which returns a subrange, while the loop still compares an iterator against it: fpga/xilinx/bitstream-reader.h:108:38: error: no match for 'operator!=' (operand types are '__gnu_cxx::__normal_iterator<const unsigned int*, ...>' and 'const std::ranges::subrange<...>') The same happened to std::find_end. Compare against the subranges' own iterators, which is what the rewritten code intended; bazel test //fpga/... passes again. Assisted-by: deepseek/deepseek-flash
treefmt (the NixCheck job) rejects four files as unformatted: assembler.cc, database.cc and arch-xc7-configuration-packet_test.cc, which the --fix commit touched, and database-parsers.cc, which has been unformatted for longer. clang-format from the flake (21.1.8, the version the check itself uses) applied to all four. No code change; nix build .#checks.x86_64-linux.treefmt passes. Assisted-by: deepseek/deepseek-flash
c90290e to
0aab4b2
Compare
* bitstream-reader.h included <optional>; the file spells the type absl::optional (from absl/types/optional.h), so clang-tidy 18 flags the standard header as unused. * injected-features.cc's PUDC_B templates split two long literals over two lines, which bugprone-suspicious-missing-comma reads as a missing comma in the array. As raw strings the literals cannot be split -- and clang-format cannot re-flow them -- and the bytes are unchanged. bazel test //fpga/... passes, nix build .#checks.x86_64-linux.treefmt exits 0, and the frames with --emit_pudc_b_pullup are identical to the binary built from main (46 non-zero frames). Assisted-by: deepseek/deepseek-flash
* assembler.cc: a const absl::Status prevented the automatic move on return (performance-no-automatic-move). The value is never modified, so drop the const. * configuration.h: the file spells absl::optional but included <optional>; the CI's clang-tidy asked for the absl header directly (misc-include-cleaner). The two clang-tidy versions disagree about absl::optional's provider: 18 (what the CI job runs) wants "absl/types/optional.h" and calls <optional> unused, while 22 resolves the alias to std::optional and wants <optional> instead. This follows 18, since that is the job's version. bazel test //fpga/... passes and nix build .#checks.x86_64-linux.treefmt exits 0. Assisted-by: deepseek/deepseek-flash
|
CI is green on Since the PR grew well past the original "gold standard" ask, here is what it actually carries:
|
This adds what you asked for in #47 after the revert in #48 — the reference chain's frames as "gold standard", and a check that
fpga-asmatches them.It does not re-apply #47, because main still carries it: the tip is
e394738andfpga/injected-features.ccis present, so #48's revert is not in effect.What's here
fpga/xilinx/testdata/reference-parity/— one directory per case withinput.fasmand, where the reference assembles it,gold.frames(the framesfasm2framesproduced, canonicalised to the frames that set a bit):xc7a35tfgg484-2xc7z010clg400-1required_features.fasmmust be emittedxc7a35tcsg324-1--emit_pudc_b_pullupxc7a35tfgg484-2xc7k325tffg900-2check.shrunsfpga-ason each input with the same database and compares the configuration bits (FPGA_AS=... check.sh). Measured:e394738(main): 5 pass, 0 fail;b234841(before Reproduce the reference implementation's output #47): 5 fail — 01 aborts on the undefined feature, the other four exit non-zero or disagree.The whole sweep, which is what set these cases: of the 21
demo-projectsdesigns that carry a.fasm, the pre-fix build produced a different configuration for 19 and no bitstream for 2; the fixed build is frame-identical to the reference on all 20 the reference can assemble (the 21st hits a database gap the reference hits too).The raw evidence — failed bitstreams, payload byte diffs between the two builds, all three stderr streams, and the one case that still disagrees with the reference — is here: https://gist.github.com/hansfbaier/20a7b7da75d2bf8fc28ad92936d620e2.
Two notes
PRJXRAY_DB_DIRand reportsskippedwithout it.bazel test: Bazel 9 has no nativesh_testand@rules_shellis not in this module graph. If you want it running in CI, either a workflow step with the toolchain devshell, or arules_shelldependency — your call, and I will wire whichever you prefer. That also covers @hzeller'stest/smoke-tests.shidea, which would then be this check plus downloading the corpus at a pinneddemo-projectsrevision.