Missing bytes in weight file: 4294967296

[svenhaile]

Hello,

SETUP - Using 1x or 4x fanless machines with Intel N150 4-core CPU (avx2), Intel UHD Graphics, of which 2x Win 11 Home, 2x Ubuntu 24.04 server, 32GB RAM each, 1GB Ethernet each

SUCCESS - A small model below 2GB installed on the Win 11 root machine works, i.e. llama3.2 1b (1.7GB) for single and multi-node

ERROR - Critical error: Missing bytes in weight file: 4294967296 for each of the remaining models mentioned in the README and installed via launch.py, i.e. Llama 3.2 >=3b, Qwen3 >=8b, and for deepseek r1 llama 8b

I welcome any suggestions

Many thanks

---

Comparison matrix of configurations and outputs for small models (rounded figures for required memory):

MODEL-Q40	RUNTYPE	NODES	MEMORY MB (root)	MEMORY MB (workers)	STATUS
llama3.2 1b (1.7 GB)	api	1	2000	-	Server URL: http://localhost:9999/v1/models - JSON
"	"	4	1300	3x 300	Server URL: http://localhost:9999/v1/models - JSON
"	chat	1	2000	-	System prompt / User prompt in CLI
"	"	4	1300	3x 300	System prompt / User prompt in CLI
"	inference	1	2000	-	Eval: b=32 t=2 t/s=23 (41ms/t) Pred: t=30 t/s=9 (114ms/t)
"	"	4	1300	3x 300	Eval: b=32 t=2 t/s=2 (560ms/t) Pred: t=30 t/s=2 (600ms/t)

MODEL-Q40	RUNTYPE	NODES	MEMORY MB (root)	MEMORY MB (workers)	STATUS
qwen3 1.7b (2.2 GB)	api	1	3100	-	std::runtime_error Missing bytes in weight file: 4294967296
"	"	4	1700	3x 500	std::runtime_error Missing bytes in weight file: 4294967296
"	chat	1	3100	-	Missing bytes in weight file: 4294967296
"	"	4	1700	3x 500	Missing bytes in weight file: 4294967296
"	inference	1	3100	-	Missing bytes in weight file: 4294967296
"	"	4	1700	3x 500	Missing bytes in weight file: 4294967296

MODEL-Q40	RUNTYPE	NODES	MEMORY MB (root)	MEMORY MB (workers)	STATUS
llama3.2 3b (3.3 GB)	api	1	125	-	Missing bytes in weight file 4294967296
"	"	4	2200	3x 700	Missing bytes in weight file: 4294967296
"	chat	1	125	-	Missing bytes in weight file 4294967296
"	"	4	2200	3x 700	Missing bytes in weight file: 4294967296
"	inference	1	125	-	Missing bytes in weight file: 4294967296
"	"	4	2200	3x 700	Missing bytes in weight file: 4294967296

---

A small model below 2GB is working fine on single and multi-node

CHAT SUCCESS

Calling dllama chat with multi-node

$ source dllama-chat-llama3_2_1b_instruct_q40_multi.sh
📄 AddBos: 35
📄 BosId: 128000 (<|begin_of_text|>)
📄 EosId: 128001 (<|end_of_text|>) 128009 (<|eot_id|>)
📄 RegularVocabSize: 128000
📄 SpecialVocabSize: 256
💡 Arch: Llama
💡 HiddenAct: Silu
💡 Dim: 2048
💡 HeadDim: 64
💡 QDim: 2048
💡 KvDim: 512
💡 HiddenDim: 8192
💡 VocabSize: 128256
💡 nLayers: 16
💡 nHeads: 32
💡 nKvHeads: 8
💡 OrigSeqLen: 131072
💡 SeqLen: 4096
💡 NormEpsilon: 0.000010
💡 RopeType: Llama3.1
💡 RopeTheta: 500000
💡 RopeScaling: f=32.0, l=1.0, h=4.0, o=8192
📀 RequiredMemory: 1266 MB   [workers each ca. 700MB]
Γ¡ò Socket[0]: connecting to <worker1>:9999 worker
Γ¡ò Socket[0]: connected
Γ¡ò Socket[1]: connecting to <worker2>:9999 worker
Γ¡ò Socket[1]: connected
Γ¡ò Socket[2]: connecting to <worker3>:9999 worker
Γ¡ò Socket[2]: connected
Γ¡ò Network is initialized
🧠 CPU: avx2
💿 Loading weights...
💿 Loaded 2/16
...
💿 Loaded 16/16
💿 Weights loaded
🚁 Network is in non-blocking mode
Γ¡É Chat template: llama3
🛑 Stop: <|end_of_text|>
🛑 Stop: <|eot_id|>
💻 System prompt (optional):

INFERENCE SUCCESS

# Inference: single-node results are identical for Win 11 Home and Ubuntu 24.04
Hello world
🔷️ Eval   64 ms Sync    0 ms | Sent     0 kB Recv     0 kB | (2 tokens)
🔶 Pred   76 ms Sync    0 ms | Sent     0 kB Recv     0 kB | !
🔶 Pred  102 ms Sync    0 ms | Sent     0 kB Recv     0 kB | This
🔶 Pred   97 ms Sync    0 ms | Sent     0 kB Recv     0 kB |  is
...
🔶 Pred   86 ms Sync    0 ms | Sent     0 kB Recv     0 kB | 20
🔶 Pred   88 ms Sync    0 ms | Sent     0 kB Recv     0 kB |  times
🔶 Pred   89 ms Sync    0 ms | Sent     0 kB Recv     0 kB |  to

Evaluation
   nBatches: 32
    nTokens: 2
   tokens/s: 31.04 (32.22 ms/tok)
Prediction
    nTokens: 30
   tokens/s: 11.00 (90.87 ms/tok)

#!/bin/sh
./dllama inference \
    --prompt "Hello world" \
    --steps 32 \
    --model models/llama3_2_1b_instruct_q40/dllama_model_llama3_2_1b_instruct_q40.m \
    --tokenizer models/llama3_2_1b_instruct_q40/dllama_tokenizer_llama3_2_1b_instruct_q40.t \
    --buffer-float-type q80 \
    --nthreads 4 \
    --max-seq-len 4096 \
    --workers <worker1>:9999 <worker2>:9999 <worker3>:9999

# Inference: multi-node results for Win 11 Home
Hello world
🔷️ Eval   24 ms Sync  766 ms | Sent   456 kB Recv  1159 kB | (2 tokens)
🔶 Pred   26 ms Sync  561 ms | Sent   228 kB Recv   579 kB | !
🔶 Pred   25 ms Sync  497 ms | Sent   228 kB Recv   579 kB |  This
🔶 Pred   23 ms Sync  518 ms | Sent   228 kB Recv   579 kB |  is
...
🔶 Pred   24 ms Sync  527 ms | Sent   228 kB Recv   579 kB |  my
🔶 Pred   22 ms Sync  555 ms | Sent   228 kB Recv   579 kB |  knowledge
🔶 Pred   41 ms Sync  442 ms | Sent   228 kB Recv   579 kB |  with

Evaluation
   nBatches: 32
    nTokens: 2
   tokens/s: 2.53 (395.10 ms/tok)
Prediction
    nTokens: 30
   tokens/s: 1.85 (540.69 ms/tok)

---

ERROR

But models larger than 2GB in size are problematic for single and multi-node

Using any ./dllama-api/inference/chat command

$ source dllama-chat-llama3_2_3b_instruct_q40_single.sh
📄 AddBos: 53
📄 BosId: 128000 (<|begin_of_text|>)
📄 EosId: 128001 (<|end_of_text|>) 128009 (<|eot_id|>)
📄 RegularVocabSize: 128000
📄 SpecialVocabSize: 256
💡 Arch: Llama
💡 HiddenAct: Silu
💡 Dim: 3072
💡 HeadDim: 128
💡 QDim: 3072
💡 KvDim: 1024
💡 HiddenDim: 8192
💡 VocabSize: 128256
💡 nLayers: 28
💡 nHeads: 24
💡 nKvHeads: 8
💡 OrigSeqLen: 131072
💡 SeqLen: 4096
💡 NormEpsilon: 0.000010
💡 RopeType: Llama3.1
💡 RopeTheta: 500000
💡 RopeScaling: f=32.0, l=1.0, h=4.0, o=8192
📀 RequiredMemory: 125 MB
🧠 CPU: avx2
💿 Loading weights...
🚨 Critical error: Missing bytes in weight file: 4294967296  <=== ERROR