中文

intel-accel-for-llm (iaxl)

iaxl uses Intel hardware accelerators to improve LLM inference performance.

Design Documentation

• IAXL Design
• KV Zip Design
• KVShrink Design

Host Setup

1. Add intel_iommu=on,sm_on iommu=pt to the kernel command line, then reboot the host:

sudo ./tools/setup_kernel_cmdline.sh
sudo reboot

2. After rebooting, install a QAT driver. Use one of the following.

   Out-of-tree driver package (default):

wget -q https://downloadmirror.intel.com/921047/QAT20.L.1.2.30-00178.tar.gz
tar xf QAT20.L.1.2.30-00178.tar.gz
./configure
make -j$(nproc)
sudo make install

Use the following commands to stop or start the out-of-tree QAT service:

adf_ctl down
adf_ctl up

In-tree driver + qatlib (kernel qat_4xxx module, virtual functions bound to vfio-pci):

sudo apt install qatlib-service libqat-dev libusdm-dev   # or build https://github.com/intel/qatlib
printf 'POLICY=0\nServicesEnabled=dc\n' | sudo tee /etc/sysconfig/qat
sudo systemctl enable qat
sudo systemctl restart qat
export IAXL_QATLIB=intree   # before sourcing setvars.sh / building

POLICY=0 gives every process one virtual function from each QAT device, which IAXL_QAT_DEVICES / KVSHRINK_QAT_DEVICES index. With POLICY=1 each process only sees device 0.

With IAXL_QATLIB=intree the build links the system libqat / libusdm found by pkg-config qatlib instead of downloading and building the out-of-tree package into _lib. Without a system qatlib (as in the dev container), it builds qatlib IAXL_QATLIB_VERSION (default 26.02.0; match the host qatmgr) into _lib, and start.sh mounts the host /run/qat.

3. Install the GDRCopy driver and configure DSA:

sudo ./tools/install_gdr_driver.sh
./tools/setup_dsa_cnt.sh

Environment Variables

Common settings in setvars.sh. Other variables in setvars.sh are advanced settings that most users do not need to change.

Category	Environment variable	Default	Description
Accelerators	IAXL_QAT_ZIP_ENABLE	1	Enable QAT compression workers (0/1)
Accelerators	IAXL_IAA_ZIP_ENABLE	0	Enable Intel IAA compression workers via Intel QPL (0/1). Can be combined with QAT; each block records whether IAA can decode it
Accelerators	IAXL_CPU_ZIP_ENABLE	0	Enable CPU compression workers (0/1). CPU compression uses a lot of CPU time, so leave it off when the QAT/IAA accelerators already provide enough throughput
Accelerators	IAXL_DSA_GD_ENABLE	0	Enable Intel DSA + GDRCopy transfers (0/1)
Compression	IAXL_KV_COMPRESSION	1	Enable DEFLATE compression of the KV cache (0/1)
Compression	IAXL_ZIP_LEVEL	1	Compression level for all zip backends, from 1 (fastest) to 9 (best ratio). QAT and CPU use the level as is. IAA hardware only supports level 1, so IAA ignores this setting. Keep the default 1: higher levels barely improve the KV cache compression ratio but make compression much slower
Compression	IAXL_KV_DATA_SHUFFLE	1	Byte-shuffle BF16 data before compression (0/1). Slightly lowers compression throughput, but improves the compression ratio and decompression throughput
Compression	IAXL_ZIP_EXPONENT_ONLY	IAXL_KV_DATA_SHUFFLE	With shuffle on, compress only the BF16 high bytes (sign and exponent) and store the low bytes raw (0/1). Improves the compression ratio and both throughputs; IAXL_KV_LOSSY_TRUNC then barely helps the ratio
vLLM	MODEL	Qwen/Qwen3-32B	Hugging Face model ID or local model path
vLLM	TP_SIZE	2	Required; number of Tensor Parallel workers. CPU, QAT, and DSA resources are configured based on this value
vLLM	DP_SIZE	1	Number of Data Parallel groups; total ranks = TP_SIZE × DP_SIZE
vLLM	VLLM_CPU_OMP_THREADS_BIND	Auto-detected from GPU NUMA topology	CPUs for vLLM, one |-separated CPU list per rank
IAXL	KVSHRINK_CPU_OMP_THREADS_BIND	Same as VLLM_CPU_OMP_THREADS_BIND	CPUs for IAXL threads, one entry per rank. Set it for CPU inference; see the vLLM CPU inference guide
IAXL	IAXL_RDMA_ENABLE	0	Use a remote KVStore daemon over RDMA (0/1); see cross-node KV Cache reuse
IAXL	IAXL_RDMA_DAEMON_NIC_IPS	unset	Daemon RDMA NIC IPs; required when IAXL_RDMA_ENABLE=1
IAXL	IAXL_RDMA_CLIENT_NIC_IPS	unset	Client RDMA NIC IPs; required on the client when IAXL_RDMA_ENABLE=1
IAXL	IAXL_OMP_THREAD_NUM	Number of compression workers	Threads driving compression. Fewer threads use fewer CPU cores

Warning

Do not enable IAXL_DSA_GD_ENABLE on GPUs that do not support P2P DMA. Keep it set to 0.

KVShrink vLLM Example

KVShrink is a vLLM V1 KV connector based on IAXL KVStore. Configure setvars.sh, then start the container:

./start.sh

setvars.sh automatically configures the CPU, QAT, and DSA resources for each rank based on the NUMA topology of the first TP_SIZE GPUs.

Inside the container, optionally install the package with pip:

pip install -e . --verbose --no-build-isolation

Start the service inside the container:

./examples/kvshrink-vllm-serve.sh

This script starts vLLM on localhost:8000, loads KVShrinkConnector, and writes logs to log.kvshrink-vllm. Use the MODEL, TP_SIZE, and per-rank CPU/QAT/DSA settings in the startup log to verify the active topology.

Open the same container from a second host terminal:

docker exec -it -w "$PWD" iaxl.vllm bash

Send a Chat Completions test request:

./tests/vllm-test.sh

To run the model on the CPU, start the container on the vLLM CPU image as described in the vLLM CPU inference guide.

KVShrink vLLM Benchmark

Keep the KVShrink vLLM service running and execute the online serving benchmark in the second container terminal:

./tests/vllm-benchmark.sh

REST API

The management API listens on localhost:18700 by default and forwards requests to each rank, except evict, which the controller handles itself.

Endpoint	Description
GET /v1/cache/status	Query cache status
POST /v1/cache/evict	Turn automatic DDR eviction on or off (on by default)
POST /v1/cache/persist	Persist cache groups to disk

curl http://localhost:18700/v1/cache/status

The scheduler evicts least-recently-used cache groups from DDR automatically once usage reaches IAXL_DDR_POOL_EVICT_HIGH_WATERMARK (0.90 by default), down to about IAXL_DDR_POOL_EVICT_LOW_WATERMARK (0.5 by default). Groups that has() reported within the last IAXL_DDR_POOL_EVICT_LEASE_S seconds (300 by default) are skipped, unless that leaves usage above IAXL_DDR_POOL_EVICT_FORCE_WATERMARK (0.8 by default): then the oldest leased groups are evicted too. Keep FORCE well above LOW, because evicting a leased group may race a load or save that has() already promised. evict takes {"enable": true|false} and returns {"auto_evict": ...}; without enable it only reports the current state. Turning it off returns after any eviction in progress has finished.

For persist, count specifies the maximum number of cache groups to process. Eviction drops unpersisted cache groups, so to keep the whole cache, pause eviction, persist, then resume:

curl -X POST http://localhost:18700/v1/cache/evict \
	-H 'Content-Type: application/json' \
	-d '{"enable":false}'
curl -X POST http://localhost:18700/v1/cache/persist \
	-H 'Content-Type: application/json' \
	-d '{"count":999999}'
curl -X POST http://localhost:18700/v1/cache/evict \
	-H 'Content-Type: application/json' \
	-d '{"enable":true}'

Cross-Node KV Cache Reuse

remote_pool supports storing KV Cache on a separate daemon node and transferring or reusing it with vLLM client nodes over RDMA. See the cross-node KV Cache guide for the topology, build and startup steps, per-rank NIC configuration, and operations.