feat: pick appropiate model quanitzation for target

.gitignore

@@ -36,3 +36,7 @@ models/*.bin
 # Environment
 .env
 .env.local
+
+# Target configuration and selector reports
+.env.topo
+selection/

README.md

@@ -11,8 +11,9 @@ This project demonstrates running large language models on CPU with inference pr
 The upstream Linux Arm64 llama.cpp server image is built with architecture-specific CPU backend variants enabled. llama.cpp can then load a backend variant that matches the Arm CPU features available at runtime.
 
 The stack includes:
+
 - llama.cpp
-- Quantized SmolLM2 135M model bundled in the image
+- Qwen3-30B-A3B-Instruct-2507 with target-aware precision selection
 - Built-in web chat interface
 - No GPU required - pure CPU inference
 
@@ -22,63 +23,62 @@ The prebuilt `ghcr.io/ggml-org/llama.cpp:server` image currently enables llama.c
 
 The Linux Arm backend variants and feature combinations are defined in upstream [`ggml/src/CMakeLists.txt`](https://github.com/ggml-org/llama.cpp/blob/ac4cddeb0dbd778f650bf568f6f08344a06abe3a/ggml/src/CMakeLists.txt#L403-L415), and the CPU server image is built with `GGML_CPU_ALL_VARIANTS=ON` in upstream [`.devops/cpu.Dockerfile`](https://github.com/ggml-org/llama.cpp/blob/ac4cddeb0dbd778f650bf568f6f08344a06abe3a/.devops/cpu.Dockerfile#L15-L19).
 
-| Backend variant | Arm features included |
-| ---------------- | --------------------- |
-| `armv8.0_1` | Baseline Armv8.0 |
-| `armv8.2_1` | Dot product |
-| `armv8.2_2` | Dot product, FP16 vector arithmetic |
-| `armv8.2_3` | Dot product, FP16 vector arithmetic, SVE |
-| `armv8.6_1` | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply |
-| `armv8.6_2` | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SVE2 |
-| `armv9.2_1` | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SME |
-| `armv9.2_2` | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SVE2, SME |
+| Backend variant | Arm features included                                                     |
+| --------------- | ------------------------------------------------------------------------- |
+| `armv8.0_1`     | Baseline Armv8.0                                                          |
+| `armv8.2_1`     | Dot product                                                               |
+| `armv8.2_2`     | Dot product, FP16 vector arithmetic                                       |
+| `armv8.2_3`     | Dot product, FP16 vector arithmetic, SVE                                  |
+| `armv8.6_1`     | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply            |
+| `armv8.6_2`     | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SVE2      |
+| `armv9.2_1`     | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SME       |
+| `armv9.2_2`     | Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SVE2, SME |
 
 ## Prerequisites
 
 1. **Arm Hardware**: An Arm system (physical or virtual).
 2. **Docker**: For container orchestration with Topo
-3. **LLM Model**: Optional when overriding the bundled default; provide a supported single-file GGUF model (e.g., Llama 3.1, Mistral, etc.)
+3. **Topo**: An experimental build supporting `configure --target` and parameter fact bindings.
+
+`topo configure --target` supplies target RAM to the build. [gguf-parser](https://github.com/gpustack/gguf-parser-go) estimates CPU memory from remote GGUF metadata, without downloading weights. The selector chooses the highest supported precision estimated to fit, from Q2_K through BF16/F16. Without target memory, selection defaults to Q4_K_M without a memory-fit check.
 
-> **Note:** `MODEL` must point to a supported single-file `.gguf` model artifact.
-> Use a Hugging Face repo ID to auto-select a CPU-friendly quantization (preferring Q4_K_M), a Hugging Face repo plus quantization suffix as `<repo>:<quantization>`, or a direct `.gguf` URL.
-> Sharded GGUFs and multimodal projector files (`mmproj`) are rejected with a clear error because this project only supports single-file text model GGUFs today.
-> Not all model repos include GGUF quantizations — look for repos with `-GGUF` in the name.
-> The selected model is downloaded when the service starts and cached inside the container.
+The default model is Qwen3-30B-A3B-Instruct-2507; other GGUF repositories with supported precisions can also be used. The model downloads at startup and is cached inside the container, including all shards when needed.
+
+Selection uses total RAM minus an OS reserve, not currently available memory. Estimates are not a fit or performance guarantee; check available RAM before deploying. Estimates include resident weights, KV cache, and compute buffers at the configured context and batch size.
 
 ## Build-Time Parameters
 
-| Parameter        | Description                                                            | Default                              |
-| ---------------- | ---------------------------------------------------------------------- | ------------------------------------ |
-| `MODEL`          | Hugging Face GGUF repo, `<repo>:<quantization>`, or direct `.gguf` URL | `unsloth/SmolLM2-135M-Instruct-GGUF` |
-| `MODEL_ENDPOINT` | Hugging Face API-compatible endpoint for repository model downloads    | `https://huggingface.co`             |
+| Parameter           | Description                                                         | Default                                           |
+| ------------------- | ------------------------------------------------------------------- | ------------------------------------------------- |
+| `MODEL`             | Hugging Face GGUF repository ID                                     | `bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF` |
+| `MODEL_ENDPOINT`    | Hugging Face API-compatible endpoint for repository model downloads | `https://huggingface.co`                          |
+| `TARGET_MEMORY_KIB` | Target RAM in KiB, populated by `configure --target`                | Q4_K_M when omitted                               |
+| `CONTEXT_LENGTH`    | Context tokens for estimation and inference                         | `4096`                                            |
 
 ## Usage
 
 The easiest way to deploy is using `topo`. Download and install `topo` from [here](https://github.com/arm/topo)
 
 ### Clone the project:
+
 ```bash
-topo clone https://github.com/Arm-Examples/topo-llama-web-ui.git
+export TOPO_EXPERIMENTAL_FEATURES=1
+topo clone https://github.com/Arm-Examples/topo-llama-web-ui.git --target <ip-address-of-target>
 ```
 
 ### Build and Deploy the project:
+
 ```bash
-cd topo-lama-web-ui
+cd topo-llama-web-ui
 topo deploy --target <ip-address-of-target>
 ```
 
 ### Common Model Selection Examples
 
 Use a different model:
-```bash
-topo deploy --target <ip-address-of-target> \
-  --arg MODEL=bartowski/Qwen_Qwen3.5-0.8B-GGUF
-```
 
-Select an exact quantization:
 ```bash
-topo deploy --target <ip-address-of-target> \
-  --arg MODEL=unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M
+topo configure MODEL=bartowski/Qwen2.5-14B-Instruct-GGUF
 ```
 
 ### Access the Chat Interface

compose.yaml

@@ -4,8 +4,10 @@ services:
     build:
       context: ./llama-inference
       args:
-        MODEL: ${MODEL:-unsloth/SmolLM2-135M-Instruct-GGUF}
+        MODEL: ${MODEL:-bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF}
         MODEL_ENDPOINT: ${MODEL_ENDPOINT:-https://huggingface.co}
+        TARGET_MEMORY_KIB: ${TARGET_MEMORY_KIB:-}
+        CONTEXT_LENGTH: ${CONTEXT_LENGTH:-4096}
     ports:
       - "8080:8080"
     healthcheck:
@@ -29,12 +31,11 @@ x-topo:
 
     The stack includes:
     - llama.cpp
-    - Quantized SmolLM2 135M default model loaded through llama.cpp
+    - Qwen3-30B-A3B-Instruct-2507 with target-aware precision selection
     - Built-in web chat interface
     - No GPU required - pure CPU inference
 
-    Perfect for demos and testing! The default SmolLM2-135M-Instruct model
-    gives the project a small ready-to-use model reference by default.
+    Topo supplies target memory; whichllm selects a precision during the build.
 
     Ideal for testing LLM workloads on Arm hardware without GPU dependencies
     while avoiding a source build during Topo Project deployment.
@@ -47,17 +48,22 @@ x-topo:
 
   args: &args
     MODEL:
-      description: |
-        Model artifact reference. Must be one of: 
-        - Hugging Face GGUF repo ID (e.g. `unsloth/SmolLM2-135M-Instruct-GGUF`)
-        - Hugging Face GGUF repo ID with quantization (e.g. `unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M`)
-        - URL to GGUF file (e.g. `https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf`)
-      default: "unsloth/SmolLM2-135M-Instruct-GGUF"
+      description: GGUF repository; defaults to Qwen3-30B-A3B-Instruct-2507 (MoE).
+      example: bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF
       hints:
         meta.type:
           - huggingface.repo-id
         model.task: text-generation
         model.format: gguf
+    TARGET_MEMORY_KIB:
+      description: Total target memory in KiB; populated by configure --target.
+      source:
+        provider: topo.memory
+        group: system-memory
+        fact: memory.total
+    CONTEXT_LENGTH:
+      description: Context tokens for both memory estimation and inference (default 4096).
+      example: "4096"
     MODEL_ENDPOINT:
       description: "Hugging Face API compatible endpoint for downloading models"
       default: "https://huggingface.co"

llama-inference/Dockerfile

@@ -1,17 +1,29 @@
+FROM --platform=$BUILDPLATFORM python:3.12-slim AS selection
+ARG BUILDARCH
+ADD --chmod=755 https://github.com/gpustack/gguf-parser-go/releases/download/v0.27.0/gguf-parser-linux-${BUILDARCH} /usr/local/bin/gguf-parser
+WORKDIR /selector
+COPY requirements.txt ./
+RUN pip install --no-cache-dir -r requirements.txt
+COPY select_model.py ./
+ARG MODEL=bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF
+ARG MODEL_ENDPOINT=https://huggingface.co
+ARG TARGET_MEMORY_KIB
+ARG CONTEXT_LENGTH=4096
+RUN python select_model.py > /model-url.txt
+
 FROM ghcr.io/ggml-org/llama.cpp:server@sha256:df320e983b2871d31fcb88165eedfcbd5156841d3d7dbd2b7ea3e85a1e1286a1
 
-ARG MODEL
-ARG MODEL_ENDPOINT
+ARG CONTEXT_LENGTH=4096
 
-ENV MODEL=${MODEL}
-ENV MODEL_ENDPOINT=${MODEL_ENDPOINT}
-ENV N_CTX=2048
+ENV N_CTX=${CONTEXT_LENGTH}
 ENV N_THREADS=8
 ENV N_BATCH=512
 ENV N_GPU_LAYERS=0
 
 WORKDIR /app
 
+COPY --from=selection /model-url.txt /app/model-url.txt
+
 COPY --chmod=755 entrypoint.sh /usr/local/bin/topo-llama-entrypoint
 
 HEALTHCHECK --interval=30s --timeout=5s --start-period=300s --retries=3 \

llama-inference/entrypoint.sh

@@ -1,26 +1,17 @@
 #!/bin/sh
 set -eu
 
-if [ -z "${MODEL:-}" ]; then
-  echo "MODEL must not be empty." >&2
-  exit 1
-fi
-
-model_ref="${MODEL}"
-
-case "$model_ref" in
-  http://*|https://*) model_option=--model-url ;;
-  *) model_option=-hf ;;
-esac
+model_ref=$(cat /app/model-url.txt)
 
 set -- \
   --host 0.0.0.0 \
   --port 8080 \
   --ctx-size "${N_CTX}" \
+  --parallel 1 \
   --threads "${N_THREADS}" \
   --batch-size "${N_BATCH}" \
   --n-gpu-layers "${N_GPU_LAYERS}" \
-  "$model_option" "$model_ref" \
+  --model-url "$model_ref" \
   "$@"
 
 exec /app/llama-server "$@"

llama-inference/requirements.txt

@@ -0,0 +1 @@
+huggingface_hub==1.30.0

llama-inference/select_model.py

@@ -0,0 +1,59 @@
+"""Select the highest supported GGUF precision estimated to fit target RAM."""
+
+import json
+import os
+import re
+import subprocess
+
+from huggingface_hub import HfApi, hf_hub_url
+
+QUANTS = ("BF16", "F16", "Q8_0", "Q6_K", "Q5_K_M", "Q4_K_M", "Q3_K_M", "Q2_K")
+FILENAME = re.compile(
+    rf"(?:^|[.-])({'|'.join(QUANTS)})(?:-00001-of-\d{{5}})?\.gguf$", re.I
+)
+
+
+def main():
+    memory = os.getenv("TARGET_MEMORY_KIB")
+    context = int(os.environ["CONTEXT_LENGTH"])
+    if context <= 0 or (memory and int(memory) <= 0):
+        raise ValueError("TARGET_MEMORY_KIB and CONTEXT_LENGTH must be positive")
+    repo, endpoint = os.environ["MODEL"], os.environ["MODEL_ENDPOINT"]
+    info = HfApi(endpoint=endpoint).model_info(repo)
+    candidates = []
+    for file in info.siblings or []:
+        match = FILENAME.search(file.rfilename)
+        if match and "mmproj" not in file.rfilename.lower():
+            quant = match[1].upper()
+            url = hf_hub_url(repo, file.rfilename, revision=info.sha, endpoint=endpoint)
+            candidates.append((QUANTS.index(quant), url))
+    candidates.sort()
+    if not memory:
+        for quality, url in candidates:
+            if QUANTS[quality] == "Q4_K_M":
+                print(url)
+                return
+        raise ValueError("Q4_K_M is unavailable")
+
+    total = int(memory) * 1024
+    # Reserve RAM for the OS and background processes.
+    budget = total - max(4 * 1024**3, min(int(total * 0.15), 32 * 1024**3))
+    for _, url in candidates:
+        result = subprocess.run([
+            "gguf-parser", "--url", url, "--json", "--gpu-layers", "0",
+            "--ctx-size", str(context), "--batch-size", "512",
+            "--ubatch-size", "512", "--parallel-size", "1", "--no-mmap",
+        ], check=True, stdout=subprocess.PIPE, text=True)
+        report = json.loads(result.stdout)
+        if context > report["architecture"]["maximumContextLength"]:
+            raise ValueError("requested context exceeds model limit")
+        # Include resident weights even though llama.cpp can memory-map them.
+        required = report["estimate"]["items"][0]["ram"]["nonuma"]
+        if required <= budget:
+            print(url)
+            return
+    raise ValueError("no supported quantization fits; use a smaller model or context")
+
+
+if __name__ == "__main__":
+    main()