Make HF generate work on transformers v4.57 and v5

[jlamypoirier]

What

Makes HuggingfaceGPTModelForCausalLM.generate() run and match HF greedy output on both transformers v4.57 (the declared floor) and v5, and enables the generate model-testing group for the models where it now passes.

Changes

can_generate() -> True on the wrapper base. transformers v5's PreTrainedModel.can_generate walks __bases__ by name and stops at any base whose name contains "PreTrainedModel"; our intermediate base hid the GenerationMixin inheritance, so the check returned False and generate() died with '... has no attribute generation_config'. Unconditional, so correct on both majors.

inner_forward absorbs cache_position (and other generate plumbing) via **kwargs. On v4.57 generate passes cache_position to forward; v5 filters it out. Ignoring it is correct on the use_cache=False path — positions are reconstructed from attention_mask, and full logits are computed with the last position selected downstream.

The wrapper honors the source HF config's generation token ids. Fast-LLM's import drops bos/eos/pad (generation metadata, not architecture), so generate never stopped at EOS. from_pretrained now applies them from the source HF config to generation_config, mirroring AutoModelForCausalLM.from_pretrained. Exposed as _apply_generation_token_ids so manually-constructed wrappers can opt in. Native Fast-LLM checkpoints (no token ids) keep the defaults.

Prediction-head logits no longer leak into the returned hidden states. inner_forward popped only the main head's logits, so multi-token-prediction heads' logits leaked into output_hidden_states. All heads' logits are now popped (extra ones discarded unless stacked), so the hidden-states count is num_blocks + prediction_heads for any head configuration.

Allowlist is_llama_config in the HF config coverage check. transformers v4 LlamaConfig carries an is_llama_config marker (dropped in v5) that Fast-LLM doesn't consume and a bare PretrainedConfig omits, so the import-boundary check rejected it — Fast-LLM could not import a real transformers-4.x Llama checkpoint. Verified on 4.57.5 that all supported converters (llama/qwen2/mistral/mixtral/mtp_llama) now report no unconsumed config keys.

Generate test group enabled for llama, mistral, mixtral, mtp_llama (verified on transformers 4.57 GPU). The CUDA-bound generate tests gain @requires_cuda so they skip on the CPU-only CI runner; test_export_for_generate stays CPU-runnable. Also fixes stale test code: vocab_size moved under embeddings, hidden-states count generalized, and test_export_for_generate uses the current DistributedTestingConfig signature.

lm_eval split into its own lm_eval testing group (it shared generate), so enabling generate doesn't pull the lm_eval tests — broken on transformers v5 — into CI.

starcoder_2 generate/convert demoted — it has no HF checkpoint format, so the export-based generate tests can't run (generate → not_implemented, was a misleading broken), and its native conversion round-trip is redundant with other models (convert → unimportant).

Left broken, with reasons

• qwen_2: matches in fp32 but a bf16/flash near-tie argmax flips within the compared horizon (numerical).
• diffusion_llama / dream: bidirectional diffusion decoding.

Out of scope (follow-up)

• lm_eval on transformers v5: import lm_eval (0.4.9) fails at class-body time (AutoModelForVision2Seq renamed to AutoModelForImageTextToText).

🤖 Generated with Claude Code