Skip to content

Warn on and normalize partial_rotary_factor declarations the DSpark modeling does not implement - #72

Open
arthurgao2003 wants to merge 1 commit into
deepseek-ai:mainfrom
arthurgao2003:fix-partial-rotary-factor-mismatch
Open

Warn on and normalize partial_rotary_factor declarations the DSpark modeling does not implement#72
arthurgao2003 wants to merge 1 commit into
deepseek-ai:mainfrom
arthurgao2003:fix-partial-rotary-factor-mismatch

Conversation

@arthurgao2003

@arthurgao2003 arthurgao2003 commented Jul 22, 2026

Copy link
Copy Markdown

Fixes #71.

Problem

The DSpark Qwen3 draft modeling applies RoPE across the full head dim (custom apply_rotary_pos_emb has no rotary/pass-through split, and Qwen3RotaryEmbedding yields full-dim cos/sin for this config), but build_draft_config starts from copy.deepcopy(target_config). A target — or a DFlash warm-start checkpoint — that declares partial_rotary_factor != 1.0 (Qwen3.5-family targets do) leaks the field into the saved draft checkpoint verbatim. The checkpoint is then self-inconsistent: weights trained with full-dim rotation, config declaring partial rotation.

Serving engines that honor the field, e.g. vLLM's dflash/dspark path, rotate head_dim * factor dims — a silent train/serve geometry mismatch. Measured on a Qwen3.5-27B-class target with a 5-layer draft warm-started from z-lab DFlash: offline AL 2.53 → vLLM serving AL ~1.4; recovers to 2.59 (matching offline) once the config declares 1.0. Full analysis in #71.

Change

  • common.py: declared_partial_rotary_factor(config) (reads rope_parameters / rope_scaling / top-level) and check_partial_rotary_factor(config, normalize=False).
  • qwen3/config.py: build_draft_config normalizes the declaration to 1.0 (with a warning) right after the deepcopy, so newly produced checkpoints describe the trained geometry.
  • qwen3/modeling.py: Qwen3DSparkModel.__init__ warns when loading a config that still declares partial RoPE — covers pre-existing checkpoints and externally generated configs.

No numerics change: training already rotated the full head dim; this only makes the saved declaration truthful and surfaces the mismatch for old checkpoints.

Testing

  • Helper unit tests: declaration read from rope_parameters / rope_scaling / top-level; normalize rewrites all present locations; clean configs stay silent and untouched.
  • Integration: real Qwen3Config with partial_rotary_factor=0.25 through build_draft_config → one normalize warning, all declarations = 1.0; Qwen3DSparkModel init on the normalized config is silent, init on a dirty config fires exactly one warning; tiny model instantiates fine on CPU.

…not implement

DSpark's Qwen3 draft modeling applies RoPE across the full head dim, but
build_draft_config deepcopies the target config, so a target (or DFlash
warm-start checkpoint) declaring partial_rotary_factor != 1.0 leaks the
field into saved draft checkpoints verbatim. Serving engines that honor
it (e.g. vLLM's dflash/dspark path) then rotate fewer dims than training
did, silently collapsing acceptance length (observed 2.53 offline ->
~1.4 in vLLM; recovers to 2.59 once the config declares 1.0).

- build_draft_config now normalizes the declaration to 1.0 with a warning
- Qwen3DSparkModel init warns when loading a config that still declares
  partial RoPE (covers pre-existing checkpoints)

Fixes deepseek-ai#71

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant