Skip to content

testdata: the reference chain's frames as gold standards, with a check - #49

Open
hansfbaier wants to merge 6 commits into
lromor:mainfrom
hansfbaier:testdata/reference-parity
Open

hansfbaier wants to merge 6 commits into
lromor:mainfrom
hansfbaier:testdata/reference-parity

Conversation

@hansfbaier

Copy link
Copy Markdown
Contributor

This adds what you asked for in #47 after the revert in #48 — the reference chain's frames as "gold standard", and a check that fpga-as matches them.

It does not re-apply #47, because main still carries it: the tip is e394738 and fpga/injected-features.cc is present, so #48's revert is not in effect.

What's here

fpga/xilinx/testdata/reference-parity/ — one directory per case with input.fasm and, where the reference assembles it, gold.frames (the frames fasm2frames produced, canonicalised to the frames that set a bit):

case part what it pins down
01 missing feature xc7a35tfgg484-2 an undefined feature must fail with a message, not terminate the process
02 zynq7 required features xc7z010clg400-1 the part's required_features.fasm must be emitted
03 PUDC_B pull-up xc7a35tcsg324-1 the pull-up is injected only with --emit_pudc_b_pullup
04 83-bit value xc7a35tfgg484-2 a value on a range wider than 64 bits resolves correctly
06 pseudo PIP on an aliased tile xc7k325tffg900-2 looked up in the tile's own database, and a no-op rather than a crash

check.sh runs fpga-as on each input with the same database and compares the configuration bits (FPGA_AS=... check.sh). Measured:

The whole sweep, which is what set these cases: of the 21 demo-projects designs that carry a .fasm, the pre-fix build produced a different configuration for 19 and no bitstream for 2; the fixed build is frame-identical to the reference on all 20 the reference can assemble (the 21st hits a database gap the reference hits too).

The raw evidence — failed bitstreams, payload byte diffs between the two builds, all three stderr streams, and the one case that still disagrees with the reference — is here: https://gist.github.com/hansfbaier/20a7b7da75d2bf8fc28ad92936d620e2.

Two notes

  • The database is not committed (hundreds of MB), so the check reads PRJXRAY_DB_DIR and reports skipped without it.
  • It is a script, not a bazel test: Bazel 9 has no native sh_test and @rules_shell is not in this module graph. If you want it running in CI, either a workflow step with the toolchain devshell, or a rules_shell dependency — your call, and I will wire whichever you prefer. That also covers @hzeller's test/smoke-tests.sh idea, which would then be this check plus downloading the corpus at a pinned demo-projects revision.

@hansfbaier

Copy link
Copy Markdown
Contributor Author

While this PR's CI ran I went through the four failures, and three of them are main's rather than this PR's. The workflow history is the clearest evidence:

when commit on main ci
11:36 Move the injected configuration out of assembler.cc (#47) failure
11:47 Revert "Reproduce the reference implementation's output" (#48) success
11:56 Move the injected configuration out of assembler.cc (again) failure
12:23 ci: clang-tidy --fix failure

So main has been red since 11:56, and the failures in my run are the same ones:

  • fpga/injected-features.cc:139/142/147 — absl::StrFormat with a runtime format string. Newer absl marks that overload unavailable, so MacOsBuild fails to compile the file and ClangTidy reports it as an error. This one is Reproduce the reference implementation's output #47's, and c90290e in this PR fixes it (absl::Substitute with $0/$1): with and without --emit_pudc_b_pullup the frames are identical to the binary built from main, and the reference-parity check passes 5/5.
  • fpga/xilinx/bitstream-reader.h:108/117 — gcc-13 cannot compare an iterator with the std::ranges::subrange that std::ranges::search returns (fails Test), and treefmt wants the file reflowed (fails NixCheck). That file's last change is 11f3fc3 (March 2025) and Reproduce the reference implementation's output #47 never touched it.

If you want, I will fix those two in a separate commit here — the bitstream-reader.h one is small (compare against the subrange's begin()), the other is running the repo's formatter over it — or leave them to a dedicated PR, since neither belongs to this one.

…t wrong

Comparing both chains over the 21 demo-projects designs that carry a .fasm
showed 19 of them assembling to a different configuration and 2 to no bitstream
at all.  This keeps the inputs behind that together with the frames
fasm2frames produced for them, and a check that runs fpga-as on the same input
and the same database and compares the configuration bits:

  check.sh      5 cases, 5 pass with the fixed assembler and 5 fail with the
                assembler built from b234841 (before the change)
  cases.tsv     part, family and expectation per case
  <case>/       input.fasm and, where the reference assembles it, gold.frames

The database is not committed (hundreds of MB), so the check reads
PRJXRAY_DB_DIR and reports skipped without it, which keeps it safe to call from
anywhere.  It is a script rather than a bazel test on purpose: Bazel 9 has no
native sh_test and @rules_shell is not in this module graph.

The raw evidence -- the failed bitstreams, the payload byte diffs between the
two builds, the stderr of all three, and the one case that still disagrees with
the reference -- is in
https://gist.github.com/hansfbaier/20a7b7da75d2bf8fc28ad92936d620e2

Assisted-by: deepseek/deepseek-flash
The pull-up templates are runtime strings, and absl::StrFormat wants a format
string that is constexpr: newer absl marks that overload unavailable, so
injected-features.cc fails to compile under clang 21 (CI's MacOsBuild job) and
clang-tidy reports it as an error.  absl::Substitute takes the format string at
run time by design, so the templates use $0/$1 placeholders and the three call
sites call Substitute.

Behaviour is unchanged: with and without --emit_pudc_b_pullup the frames are
identical to the binary built from main (44 and 46 non-zero frames
respectively), and the reference-parity check passes 5/5.

Assisted-by: deepseek/deepseek-flash
The clang-tidy --fix commit (957dafc) rewrote

  std::search(words_.cbegin(), words_.cend(), kWcfgCmd.cbegin(), kWcfgCmd.cend())

into std::ranges::search(words_, kWcfgCmd), which returns a subrange, while the
loop still compares an iterator against it:

  fpga/xilinx/bitstream-reader.h:108:38: error: no match for 'operator!='
    (operand types are '__gnu_cxx::__normal_iterator<const unsigned int*, ...>'
     and 'const std::ranges::subrange<...>')

The same happened to std::find_end.  Compare against the subranges' own
iterators, which is what the rewritten code intended; bazel test //fpga/...
passes again.

Assisted-by: deepseek/deepseek-flash
treefmt (the NixCheck job) rejects four files as unformatted: assembler.cc,
database.cc and arch-xc7-configuration-packet_test.cc, which the --fix commit
touched, and database-parsers.cc, which has been unformatted for longer.

clang-format from the flake (21.1.8, the version the check itself uses) applied
to all four.  No code change; nix build .#checks.x86_64-linux.treefmt passes.

Assisted-by: deepseek/deepseek-flash
@hansfbaier
hansfbaier force-pushed the testdata/reference-parity branch from c90290e to 0aab4b2 Compare September 25, 2026 13:09
* bitstream-reader.h included <optional>; the file spells the type
  absl::optional (from absl/types/optional.h), so clang-tidy 18 flags the
  standard header as unused.
* injected-features.cc's PUDC_B templates split two long literals over two
  lines, which bugprone-suspicious-missing-comma reads as a missing comma in the
  array.  As raw strings the literals cannot be split -- and clang-format cannot
  re-flow them -- and the bytes are unchanged.

bazel test //fpga/... passes, nix build .#checks.x86_64-linux.treefmt exits 0,
and the frames with --emit_pudc_b_pullup are identical to the binary built from
main (46 non-zero frames).

Assisted-by: deepseek/deepseek-flash
* assembler.cc: a const absl::Status prevented the automatic move on return
  (performance-no-automatic-move).  The value is never modified, so drop the
  const.
* configuration.h: the file spells absl::optional but included <optional>; the
  CI's clang-tidy asked for the absl header directly (misc-include-cleaner).

The two clang-tidy versions disagree about absl::optional's provider: 18 (what
the CI job runs) wants "absl/types/optional.h" and calls <optional> unused,
while 22 resolves the alias to std::optional and wants <optional> instead.  This
follows 18, since that is the job's version.

bazel test //fpga/... passes and nix build .#checks.x86_64-linux.treefmt exits 0.

Assisted-by: deepseek/deepseek-flash
@hansfbaier

Copy link
Copy Markdown
Contributor Author

CI is green on 4d75e90 — ClangTidy, MacOsBuild, Test and NixCheck all pass.

Since the PR grew well past the original "gold standard" ask, here is what it actually carries:

commit what
9b8b9b8 fpga/xilinx/testdata/reference-parity/ — inputs, the reference's frames, and check.sh: 5 cases, 5 pass against this tree and 5 fail against the assembler built from b234841
d77cbc5 injected-features.cc: the PUDC_B templates go through absl::Substitute — the runtime format string was a hard error for MacOsBuild and clang-tidy
4977f56 bitstream-reader.h: compare against the subrange iterators the clang-tidy --fix rewrite (957dafc) requires
0aab4b2 run clang-format over assembler.cc, database.cc, database-parsers.cc, arch-xc7-configuration-packet_test.cc — treefmt rejected them
6968bf0 drop the unused <optional>, and make the PUDC_B template literals raw strings so bugprone-suspicious-missing-comma stops reading a split literal as a missing comma
4d75e90 drop the const that blocked the automatic move in assembler.cc, and include absl/types/optional.h in configuration.h

main (957dafc) is still red for the same reasons — merging this should turn it green. Nothing here re-applies #47; main already carries it, and the revert in #48 is not in effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant