feat: support video nsa#463
Conversation
|
/test |
| auto batched_grids = grid_tensors.size() == 1 ? grid_tensors.front() : infinicore::op::cat(grid_tensors, 0); | ||
| auto batched_vision_hidden = visual_->forward(batched_pixels, batched_grids); | ||
|
|
||
| std::vector<size_t> visual_token_ranges; |
There was a problem hiding this comment.
再确认下visual_token_ranges变量是不是得在python/infinilm/processors/videonsa_processor.py中计算好，传递过来得，而不是在c++中to_cpu现算
There was a problem hiding this comment.
如果是的话，得修改bind的Input结构体，最后的位置新增一个变量。
| def normalize_hf_config_for_infinilm(config_dict, model_path): | ||
| model_type = config_dict.get("model_type") | ||
|
|
||
| if model_type == "qwen2_5_vl" and config_dict.get("architectures") == [ |
There was a problem hiding this comment.
这是啥意思，为什么要吧"model_type"从qwen2_5_vl修改为videonsa。
| normalized["text_config"] = text_config | ||
| return normalized | ||
|
|
||
| return config_dict |
There was a problem hiding this comment.
这个模型适配了的话，是不是之后qwen2_5_vl模型的适配，能复用csrc/models/videonsa/文件夹中的绝大部分文件
pengcheng888
left a comment
pengcheng888
left a comment
There was a problem hiding this comment.
修改后的代码，（1）给出测试命令和测试截图；（2）已经有多模态模型也能跑
wooway777
force-pushed
the
feat/support-video-nsa
branch
from
443e44c to
92d2ee4
Compare
|
/retest |
|
⛔ Only repository members can run |
wooway777
force-pushed
the
feat/support-video-nsa
branch
2 times, most recently
from
4121275 to
24c8323
Compare
|
/test |
|
✅ Started CI workflow run 28448940484 for commit |
wooway777
force-pushed
the
feat/support-video-nsa
branch
from
24c8323 to
04ba524
Compare
|
/retest |
|
✅ Started CI workflow run 28498414541 for commit |
wooway777
force-pushed
the
feat/support-video-nsa
branch
from
04ba524 to
6a5ea37
Compare
| return cpu->to(device); | ||
| } | ||
|
|
||
| std::pair<infinicore::Tensor, infinicore::Tensor> build_mrope_cache(size_t max_seq_len, |
There was a problem hiding this comment.
这一部分先临时这么搞，泽众会做一个通用的rope模块，等他那边确定了再替换这部分，从而避免在推理逻辑中创建sin cos表
wooway777
force-pushed
the
feat/support-video-nsa
branch
from
6a5ea37 to
d9f14e6
Compare
3 participants
CUDA_VISIBLE_DEVICES=4,5 python examples/test_infer.py --model=/data-aisoft/mechdancer/models/Enxin_VideoNSA/ --image=/data-aisoft/pepe/images/im_sh_landing.jpg --prompt="describe the image" --enable-paged-attn --attn=flash-attn目前看起来开了nsa会有精度损失，进而在常规输出后会有重复。这个后续再看能不能修吧，目前感觉不是一口气能无痛做好的。
bench_videonsa.py主要是给指标用的，不太需要在意细节。
MiniCPM V 2.6
9g8b
