Make the JIT cache key independent of the install include path - #388
Open
matteso1 wants to merge 1 commit into
Open
Make the JIT cache key independent of the install include path#388matteso1 wants to merge 1 commit into
matteso1 wants to merge 1 commit into
Conversation
The key embedded the absolute -I<library include> token via flags, so byte-identical kernels re-JIT and duplicate cache entries whenever the install prefix differs. That directory's header content is already keyed through the include hash embedded in the code string; the compiler version signature, every behavioral flag, and every other include path (CUDA headers are not content-hashed) stay in the key. Compile commands are unchanged; only the key computation differs. Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
This was referenced Jul 20, 2026
31 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The kernel cache key embeds the absolute
-I{library_include_path}token viaflags, so a byte-identical kernel re-JITs and duplicates cache entries whenever the install prefix differs (venv rename, container build-layer vs runtime layout, moved NFS mount). That defeats sharing aDG_JIT_CACHE_DIRacross layouts.This strips that one token from the key (
strip_flag_token, exact match, key computation only; the compile command andDG_JIT_DEBUGoutput are unchanged). It stays safe because that directory's header content is already keyed through the include hash embedded incode(IncludeParser::get_hash_value); everything else stays keyed: the compiler-versionsignature, every behavioral flag, and every other-I(NVRTC'scuda_home/includeheaders are not content-hashed, so their path stays). Since the key changes, existing entries re-JIT once under it.This PR follows simon-mo's request on the open vllm-project/vllm#48190 to propose the relocation fix upstream instead of carrying it as a vendored patch. The vendored variant proposed there also drops the version field; on H100, relocating a prebuilt cache across install layouts turned a guaranteed miss into a hit and cut 93.7s from a Qwen3-30B-A3B-FP8 cold start (n=1). This PR deliberately keeps the version field keyed.
Related but distinct: #301/#302 (JIT cache concurrency), #333 (wheel portability).