FastPLMs 1.0 repository and compliance upgrade

.dockerignore

@@ -1,18 +1,38 @@
-# Ignore everything by default
+# The build context is an explicit allowlist. Runtime stages never receive the
+# upstream parity oracles; reference stages copy only their selected checkout.
 *
 
-# Keep Python source files anywhere in the tree
-!**/*.py
+!.dockerignore
+!.gitmodules
+!kernels.lock
+!README.md
+!LICENSE
+!THIRD_PARTY_NOTICES.md
 
-# Keep dependency manifest used during build
-!requirements.txt
+!requirements/
+!requirements/**
+!LICENSES/
+!LICENSES/**
+!src/
+!src/**
+!tests/
+!tests/**
+!benchmarks/
+!benchmarks/**
+!tools/
+!tools/**
+!docker/
+!docker/**
+!docs/
+!docs/**
+!model_cards/
+!model_cards/**
 
-# Keep official submodules (needed for pip install -e during build)
-!official/**
+!vendor/
+!vendor/README.md
 
-# Keep non-Python files needed by models
-!**/*.json
-!**/*.txt
-!**/*.md
-!Dockerfile
-!.dockerignore
+**/.git
+**/.git/**
+**/__pycache__
+**/*.py[cod]
+**/.pytest_cache

.gitattributes

@@ -0,0 +1,4 @@
+* text=auto eol=lf
+# Preserve byte-exact third-party legal text without treating its upstream
+# trailing spaces as project defects.
+LICENSES/ankh/LICENSE.md whitespace=-trailing-space

.gitignore

@@ -1,27 +1,53 @@
+# Python
+__pycache__/
+*.py[cod]
+*.so
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+.venv/
 
-*.pyc
-*.png
-!docs/assets/*.png
-*.pth
-*.pt
-*.safetensors
-*.json
-!e1_fastplms/*.json
-*.bin
-.qodo
+# Build and run outputs
+build/
+dist/
+/artifacts/
+results/
+.cache/
 *.db
+*.db-shm
+*.db-wal
 *.nbc
 *.nbi
+
+# Checkpoint and tensor payloads
+*.bin
 *.ckpt
-/results_classification_lora
-/results_regression_lora
-/testing/results
+*.pth
+*.pt
+*.safetensors
+!tests/goldens/**/*.safetensors
+
+# Local credentials and workstation state
+.env
+.env.*
+!.env.example
+*.key
+*.pem
+*.p12
+*.pfx
+.qodo/
+
+# Local planning and scratch material
 draft.md
-/dplm
-/E1
-/.cache
-/esm
-/testing/__pycache__
-/github_issues
-/internal
-/marketing
+github_issues/
+internal/
+marketing/
+/.codex-*
+/.codex_*
+/.claude
+/tmp
+/output
+*.log
+.secrets.env

.gitmodules

@@ -1,12 +1,27 @@
-[submodule "official/boltz"]
-	path = official/boltz
+[submodule "vendor/upstream/ankh"]
+	path = vendor/upstream/ankh
+	url = https://github.com/agemagician/Ankh.git
+[submodule "vendor/upstream/biohub-esm"]
+	path = vendor/upstream/biohub-esm
+	url = https://github.com/Biohub/esm.git
+[submodule "vendor/upstream/biohub-transformers"]
+	path = vendor/upstream/biohub-transformers
+	url = https://github.com/Biohub/transformers.git
+[submodule "vendor/upstream/boltz"]
+	path = vendor/upstream/boltz
 	url = https://github.com/jwohlwend/boltz.git
-[submodule "official/e1"]
-	path = official/e1
-	url = https://github.com/Profluent-AI/E1.git
-[submodule "official/dplm"]
-	path = official/dplm
+[submodule "vendor/upstream/dplm"]
+	path = vendor/upstream/dplm
 	url = https://github.com/bytedance/dplm.git
-[submodule "official/esm"]
-	path = official/esm
-	url = https://github.com/Biohub/esm.git
+[submodule "vendor/upstream/e1"]
+	path = vendor/upstream/e1
+	url = https://github.com/Profluent-AI/E1.git
+[submodule "vendor/upstream/fair-esm"]
+	path = vendor/upstream/fair-esm
+	url = https://github.com/facebookresearch/esm.git
+[submodule "vendor/upstream/openfold"]
+	path = vendor/upstream/openfold
+	url = https://github.com/aqlaboratory/openfold.git
+[submodule "vendor/upstream/protein-ttt"]
+	path = vendor/upstream/protein-ttt
+	url = https://github.com/anton-bushuiev/ProteinTTT.git

AGENTS.md

@@ -1,29 +1,157 @@
 # FastPLMs
 
-## Purpose and Sources
+## Purpose
 
-FastPLMs provides Hugging Face-compatible protein language and structure models under `fastplms/`.
+FastPLMs maintains the runtime source uploaded to Hugging Face protein language
+and structure model repositories. This repository is a source, test, artifact,
+and dependency workspace, not an installable Python distribution. Changes must
+preserve biological conventions, Transformers behavior, reproducibility, legal
+provenance, and the evidence boundary of each model family.
 
-- `docs/architecture.md` and `docs/models.md`: package and model-family contracts
-- `docs/embedding_api.md`: shared embedding interface
-- `docs/attention_backends.md`: backend-specific behavior
-- `docs/testing.md`: per-family images, markers, and parity workflow
+Start with [README.md](README.md) for user-facing behavior and
+[docs/README.md](docs/README.md) for documentation routing.
 
-## Architectural Invariants
+## Sources of truth
 
-- Model and config classes remain compatible with Transformers auto classes and `trust_remote_code=True`.
-- `fastplms/embedding_mixin.py` is the shared sequence embedding API.
-- Tokenizer-mode families accept token IDs and attention masks. E1 sequence mode has no tokenizer and must retain its native raw-sequence preparation path.
-- `testing/conftest.py` is the authoritative model registry, and `testing/test_parity.py` is the strict parity suite.
-- Use per-family Docker images for native dependency parity. Respect the `gpu`, `slow`, `large`, and `structure` markers before running expensive suites.
+- `src/fastplms/models.toml`: model IDs, revisions, files, AutoClasses,
+  tokenizer modes, state transformations, backends, precision, licenses, and
+  release tiers.
+- `src/fastplms/registry.py`: typed parsing and validation of the manifest.
+- `docs/architecture.md` and `docs/models.md`: runtime-source and model-family
+  contracts.
+- `docs/embedding_api.md`: shared ordered embedding interface and persistence.
+- `docs/attention_backends.md`: backend names, dtype constraints, masks, and
+  parity boundaries.
+- `docs/testing.md`: candidate/reference stages, markers, and parity workflow.
+- `tests/parity/`: strict model and tokenizer comparisons.
 
-## Canonical Commands
+Do not infer a model contract from an older README, an unpinned Hub card, or an
+unused code path when the manifest or current tests say otherwise.
+
+## Repository boundaries
+
+- `src/fastplms/` contains runtime source copied into Hugging Face artifacts.
+- `vendor/upstream/` contains pinned official repositories used as parity
+  oracles. Runtime code must not import from this directory.
+- `tests/` contains unit, integration, parity, structure, and release checks.
+- `tools/` contains artifact, conversion, remote, and maintenance workflows.
+- `examples/` contains runnable research and training examples. Keep examples
+  directly in this directory rather than creating a tutorial subtree.
+- `model_cards/` contains generated checkpoint cards.
+- `LICENSES/` contains distributable third-party legal texts and provenance.
+
+Do not place license files, model cards, or READMEs beside runtime model
+modules. Do not hand-edit generated model cards or
+`docs/generated/support.md`; update the manifest or renderer and regenerate.
+
+## Architectural invariants
+
+- Model and configuration classes remain compatible with Transformers auto
+  classes and `trust_remote_code=True`.
+- `src/fastplms/embeddings/` is the shared sequence embedding API.
+- Tokenizer-mode families accept token IDs and attention masks. E1 has no
+  tokenizer and must retain its native raw-sequence preparation path.
+- Structure families retain native chain, residue, atom, ligand, nucleic-acid,
+  and MSA semantics where applicable.
+- A requested attention backend either executes the named implementation or
+  raises. Never add a silent fallback.
+- Official repositories are isolated references, not build inputs for runtime
+  source. Production imports must not change `sys.path`, download code, compile
+  a kernel, initialize a model, or mutate global Torch state.
+- State transformations are named, deterministic, and covered by exact tests.
+- Boltz2 remains provisional until its declared native end-to-end equivalence
+  limits pass. Do not broaden its claims from partial contracts.
+
+## Common workflows
+
+Initialize official sources:
 
 ```bash
 git submodule update --init --recursive
-./build_images.sh esm2
-docker run --rm --gpus all --ipc=host -v ${PWD}:/workspace fastplms-esm2 \
-  python -m pytest /workspace/testing/test_parity.py -k esm2 -v
 ```
 
-Always pass `--ipc=host` to Dockerized PyTorch runs.
+Run portable release checks on the declared remote environment:
+
+```bash
+python -m tools.remote \
+  --host user@gpu-host \
+  --identity /path/to/key \
+  --suite check
+
+python -m tools.remote \
+  --host user@gpu-host \
+  --identity /path/to/key \
+  --suite compliance
+```
+
+Build the candidate and one isolated reference image:
+
+```bash
+sudo docker buildx bake \
+  -f docker/docker-bake.hcl \
+  candidate reference-esm2 \
+  --load
+```
+
+Run focused candidate tests only when their required reference results already
+exist:
+
+```bash
+sudo docker compose -f docker/compose.yaml run --rm candidate \
+  python -m pytest tests/parity -k esm2 -v
+```
+
+Compose supplies `ipc: host` for its services. Pass `--ipc=host` to raw
+Dockerized PyTorch runs. Respect the `gpu`, `slow`, `large`, and `structure`
+markers before running expensive suites.
+
+Regenerate and check documentation:
+
+```bash
+PYTHONPATH=src python -m tools.artifacts.generate_docs
+PYTHONPATH=src python -m tools.artifacts.generate_docs --check
+python -m pytest tests/release/test_documentation.py \
+  tests/release/test_model_card_licenses.py -v
+```
+
+Build a local Hub artifact:
+
+```bash
+PYTHONPATH=src python -m tools.artifacts.build \
+  esm2_150m \
+  /cache/fast-snapshot \
+  --tokenizer-dir /cache/official-tokenizer-snapshot \
+  --output-root dist/hub
+```
+
+Preview an add-only Hub update that excludes checkpoint weights:
+
+```bash
+PYTHONPATH=src python -m tools.artifacts.publish \
+  --files-only \
+  --artifact-root dist/hub \
+  --dry-run
+```
+
+Remove `--dry-run` only after reviewing every planned path. The publisher must
+select every manifest model when no positional model IDs are provided; model
+IDs restrict the operation to that explicit subset. It must remain
+manifest-scoped, add-only, protected by the remote parent commit, and free of
+token command-line arguments. It must never upload weight-shaped paths, create
+repositories, delete remote files, or publish complete-artifact attestations
+during a files-only update.
+
+## Change policy
+
+- Inspect the manifest, family code, tests, and current docs before changing a
+  biological or model-facing contract.
+- Keep changes focused and preserve unrelated work in a dirty tree.
+- Add or update tests for public behavior, conversion rules, file identities,
+  generated output, and fail-closed paths.
+- Use repository-native verification proportional to risk. Do not run large
+  GPU or structure suites without the requested environment and authority.
+- Record measured results with the environment, dtype, backend, sequence panel,
+  and threshold. Do not turn an implementation detail into a parity, speed, or
+  biological claim.
+- Never inspect or print credential files. Pass credential paths opaquely to
+  the trusted command that needs them.