Progressive merkleization support + Enforce soft unmarshal limits

.github/workflows/test.yaml

@@ -11,28 +11,48 @@ on:
       - main
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v5
+
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v5
+      with:
+        go-version-file: 'go.mod'
+
+    - name: Run golangci-lint
+      uses: golangci/golangci-lint-action@v9
       with:
-        go-version: 1.19
-    - name: Download golangci-lint
-      run: wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s latest
-    - name: Lint
-      run: ./bin/golangci-lint run --config .golangci.yml
+        version: v2.14
 
   test:
     runs-on: ubuntu-latest
     needs: lint
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v5
+
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v5
       with:
-        go-version: 1.19
+        go-version-file: 'go.mod'
+
     - name: Test
       run: go test -v ./...
+
+  tidy:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Go
+      uses: actions/setup-go@v5
+      with:
+        go-version-file: 'go.mod'
+
+    - name: Tidy checker
+      run: go mod tidy -diff

.golangci.yml

@@ -1,42 +1,39 @@
-# This file configures github.com/golangci/golangci-lint.
-
+version: "2"
 run:
-  timeout: 20m
   tests: true
-  # default is true. Enables skipping of directories:
-  #   vendor$, third_party$, testdata$, examples$, Godeps$, builtin$
-  skip-dirs-use-default: true
-
 linters:
-  disable-all: true
+  default: none
+  # goconst: repeated literals in test tables read better inline.
   enable:
-    - goconst
-    - goimports
-    - gosimple
+    - bidichk
+    - durationcheck
+    # - goconst 
     - govet
     - ineffassign
     - misspell
-    - unconvert
-    - typecheck
-    # - unused
     - staticcheck
-    - bidichk
-    - durationcheck
-    - exportloopref
+    - unconvert
     - whitespace
-
-    # - structcheck # lots of false positives
-    # - errcheck #lot of false positives
-    # - contextcheck
-    # - errchkjson # lots of false positives
-    # - errorlint # this check crashes
-    # - exhaustive # silly check
-    # - makezero # false positives
-    # - nilerr # several intentional
-
-linters-settings:
-  gofmt:
-    simplify: true
-  goconst:
-    min-len: 3 # minimum length of string constant
-    min-occurrences: 6 # minimum number of occurrences
+  exclusions:
+    generated: lax
+    presets:
+      - comments
+      - common-false-positives
+      - legacy
+      - std-error-handling
+    paths:
+      - third_party$
+      - builtin$
+      - examples$
+formatters:
+  enable:
+    - goimports
+  settings:
+    gofmt:
+      simplify: true
+  exclusions:
+    generated: lax
+    paths:
+      - third_party$
+      - builtin$
+      - examples$

README.md

@@ -1,26 +1,84 @@
 methodical-ssz
 --------------
 
-This tool generates code that uses the [prysm fork](https://github.com/prysmaticlabs/fastssz/) of the [fastssz api](https://github.com/ferranbt/fastssz/) (with bindings for [go-hashtree](https://github.com/prysmaticlabs/gohashtree) coming very soon!), for marshaling, unmarshaling and merkleization of go types.
+This tool generates code, for marshaling, unmarshaling and merkleization (with bindings for [go-hashtree](https://github.com/prysmaticlabs/gohashtree)) of go types. The original generated code patterns and library methods were based on code generated by [fastssz](https://github.com/ferranbt/fastssz). Many thanks to Ferran and the other contributors!
 
 Generate ssz methodsets
 -----------------------
 
  The source types are parsed using `go/types`, which locates the source code for a given package path through go's local package discovery utilities, so a current limitation is that you need to fetch the package to your go package tree before using the tool. This can be done with go get:
 ```
-go get github.com/prysmaticlabs/prysm/v3@v3.2.1
+go get github.com/prysmaticlabs/prysm/v7@latest
 ```
 
 Once this is done, code generation can be run against go types in the desired package:
 ```
-go run ./cmd/ssz gen --output=beacon-state.ssz.go --type-names BeaconState,BeaconStateAltair,BeaconStateBellatrix github.com/prysmaticlabs/prysm/v3/proto/prysm/v1alpha1
-Generating methods for github.com/prysmaticlabs/prysm/v3/proto/prysm/v1alpha1/BeaconState
-Generating methods for github.com/prysmaticlabs/prysm/v3/proto/prysm/v1alpha1/BeaconStateAltair
-Generating methods for github.com/prysmaticlabs/prysm/v3/proto/prysm/v1alpha1/BeaconStateBellatrix
+go run ./cmd/ssz gen --output=beacon-state.ssz.go --type-names BeaconState,BeaconStateAltair,BeaconStateBellatrix github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1
+Generating methods for github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1/BeaconState
+Generating methods for github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1/BeaconStateAltair
+Generating methods for github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1/BeaconStateBellatrix
 ```
 
 The generated file can then be placed in a local directory for the source package so that the generated methods become part of the methodset of the source type.
 
+Generator config yaml
+---------------------
+
+The `gen` subcommand accepts a `--config` flag pointing at a yaml file, as an alternative to the `--package`/`--type-names` flags. The config names the source package and the types to generate, and is also where per-type codegen customization lives. (Note this is a different file from the spectest mapping yaml described below — projects typically keep several generator configs, for example one per fork.)
+
+The minimal config is equivalent to the cli flags:
+```yaml
+package: github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1
+types:
+  - name: BeaconState
+  - name: BeaconStateAltair
+```
+
+```
+go run ./cmd/ssz gen --config=beacon-state.yaml --output=beacon-state.ssz.go
+```
+
+ProgressiveContainer
+====================
+
+A type entry can be marked as an SSZ `ProgressiveContainer` (see the [progressive merkleization spec](https://github.com/ethereum/consensus-specs/blob/master/ssz/simple-serialize.md)) by adding a `progressive` key:
+```yaml
+package: github.com/example/mytypes
+types:
+  - name: IndexedAttestation
+    progressive: {}
+```
+
+A marked type's merkleization methods are generated as `ProgressiveHashTreeRoot()` and `ProgressiveHashTreeRootWith(hh *ssz.Hasher)`, with the standard `HashTreeRoot`/`HashTreeRootWith` generated as thin wrappers that call the progressive forms. Callers — including the generated methods of enclosing container types — always use the standard names, and marked types redirect internally, so nesting progressive and regular containers requires no further configuration. Serialization (`SizeSSZ`/`MarshalSSZ`/`UnmarshalSSZ`) is unaffected: the spec serializes progressive types identically to their non-progressive counterparts.
+
+An empty `progressive: {}` object derives the container's `active_fields` bitvector as all-ones, with one position per Go struct field. When fields have been removed from (or reserved in) the container's schema across forks, the gaps are expressed with `inactive_indices` — a list of 0-indexed positions in the bitvector forced to 0. The final bitvector length is the Go field count plus the number of inactive indices:
+```yaml
+types:
+  - name: ExecutionPayloadHeader
+    progressive:
+      # the type declares 3 go fields; positions 1 and 3 are reserved,
+      # so active_fields = [1, 0, 1, 0, 1]
+      inactive_indices: [1, 3]
+```
+
+The spec's validity rules are enforced at generation time: a progressive container must have at least one field, `active_fields` is limited to 256 positions, must not end in an inactive position, and inactive indices must be unique and in range.
+
+ProgressiveList, ProgressiveBitlist, ProgressiveByteList
+========================================================
+
+Progressive collections have no spec limit, so they are also declared in the config rather than struct tags: a type entry's `fields` key maps Go struct field names to an object with a `type` key.
+```yaml
+types:
+  - name: BeaconState
+    progressive: {}
+    fields:
+      Balances:          {type: "ProgressiveList"}     # go type: []uint64
+      ParticipationBits: {type: "ProgressiveBitlist"}  # go type: go-bitfield Bitlist
+      ExtraData:         {type: "ProgressiveByteList"} # go type: []byte
+```
+
+`ProgressiveList` is the progressive merkleization form of the ssz `List` type (`ProgressiveByteList` is the byte-slice special case, equivalent to the spec's alias for `ProgressiveList[byte]`). `ProgressiveBitlist` applies to a named bitlist type backed by a byte slice, like `github.com/OffchainLabs/go-bitfield`'s `Bitlist`. Hashing uses the spec's `merkleize_progressive` with a length mixin, with no limit. ProgressiveList family fields still use the `ssz-size`/`ssz-max` struct tags, so element dimensions can be resolved, and the field's `ssz-max` (present on `ProgressiveBitlist` fields optionally) is enforced at unmarshal time as a defensive check: generated `UnmarshalSSZ` rejects inputs exceeding it, exactly like the non-progressive form, so malicious inputs cannot trigger unbounded allocation. Progressive collection fields may be used in any generated type; the containing type does not need to be marked `progressive`.
+
 Generate spectests for a package
 --------------------------------
 
@@ -39,7 +97,7 @@ yaml config
 
 The spectest generation tool needs a mapping between go types and consensus spec container types. This mapping is described as a yaml config file. The mappings for prysm types are committed to the repo in a go `testdata` fixture directory, in `specs/testdata/prysm.yaml`. The following snippet illustrates the format:
 ```
-package: github.com/prysmaticlabs/prysm/v3/proto/prysm/v1alpha1
+package: github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1
 preset: mainnet
 defs:
   - fork: phase0
@@ -74,3 +132,38 @@ Run the spectest like normal go tests:
 go test ./generated
 ok  	github.com/OffchainLabs/methodical-ssz/generated	1.003s
 ```
+
+gen-spectest: generate spectests in your own module
+====================================================
+
+The `spectest` subcommand above runs from inside this repository and builds a self-contained package (copying the type definitions and regenerating their methodsets), which requires methodical-ssz to depend on the package being tested. When your project already consumes methodical-ssz, that creates an awkward circularity — and the tests exercise a copy rather than the methodsets you ship.
+
+The `gen-spectest` subcommand is the alternate mode for that workflow: run it from the directory of the module that owns the types. It performs no package loading and no code generation — the yaml config (same format as above) supplies the type names and the types package import path, and the tarball supplies the fixtures — so the tool needs no access to your types at all. The generated tests import your package and exercise the ssz methodsets already committed in your tree (produced by `gen`).
+
+```
+go run github.com/OffchainLabs/methodical-ssz/cmd/ssz gen-spectest \
+  --release-uri=file://$PWD/mainnet-v1.3.0-rc.2.tar.gz \
+  --config=./prysm.yaml \
+  --output=./proto/prysm/v1alpha1/spectest
+```
+
+The output directory receives a test-only package (`package spectest` by default, override with `--package-name`) containing `methodical_test.go` and the fixture files under `testdata/`. Run it like any go test:
+```
+go test ./proto/prysm/v1alpha1/spectest
+```
+
+Since the fixtures are large and reproducible from the release tarball, the output directory is a good candidate for your .gitignore.
+
+When the types span multiple Go packages (for example most consensus types in `proto/prysm/v1alpha1` but the execution-layer containers in `proto/engine/v1`), a type entry can override the top-level `package:` with its own `package` key, and a single generated test package will import each used package under a deterministic alias:
+```yaml
+package: github.com/prysmaticlabs/prysm/v7/proto/prysm/v1alpha1   # default
+preset: mainnet
+defs:
+  - fork: bellatrix
+    types:
+      - name: BeaconBlock
+        type_name: BeaconBlockBellatrix          # default package
+      - name: ExecutionPayload
+        package: github.com/prysmaticlabs/prysm/v7/proto/engine/v1 # override
+```
+A blank `package` means the top-level default; fork inheritance carries the package along with the type name, so a later fork can move a type to a different package. Only packages whose types matched a fixture are imported.

cmd/ssz/commands/all.go

@@ -0,0 +1,5 @@
+package commands
+
+import "github.com/urfave/cli/v2"
+
+var All = []*cli.Command{Generate, IR, Tests, GenSpectest}