Repository navigation
Conversation
Preserve video frame descriptors through row selection, coalesce frame decoding across a batch, and expose backend selection for training windows. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-6-astra Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 181/181 AI-Contributed/UT: 103/103
JingsongLi
left a comment
There was a problem hiding this comment.
Found one reproducible memory issue in the video decoding path; details inline.
| def _decode_video_frames(decoder, frame_indices, unused_rows): | ||
| get_frames_at = getattr(decoder, "get_frames_at", None) | ||
| if get_frames_at is not None: | ||
| return get_frames_at(indices=frame_indices).data |
There was a problem hiding this comment.
[P2] Give each decoded frame independent storage
get_frames_at(...).data is a batched Tensor, so the collator assigns views of its storage to individual rows. _sample() then calls copy.deepcopy() separately for each requested frame (lines 303–305), which copies the entire backing batch storage for every view. With no column_transforms, a real PyAV read of 32 three-frame windows (34 decoded rows) returned 96 independent storages of 4,896 bytes each, although each 8×6 RGB frame contains only 144 bytes: 470,016 retained bytes instead of 13,824 (34×). torch.stack releases those oversized copies after each transform, but still incurs the full batch copies. This substantially increases memory and copying costs for video-training batches. Please clone each frame before passing it to the collator, or use Tensor.clone() instead of deepcopy in _sample(), and add a regression check for the returned frames' storage size. Per-frame cloning reduced this reproduction to 1× storage with identical frame values.
Summary
ContiguousWindowDatasetcurrently rejectsvideo-frame-fieldcolumns, so callers cannot train directly from video-backed windows. This change keeps video descriptors through row selection and decodes the selected frames only after overlapping window reads have been coalesced.Changes
video-frame-fieldcolumns inContiguousWindowDatasetwith the existing PyAV and TorchCodec decoder path.blob_parallelismsetting.video_backendselection toScanQuery.to_contiguous_window_dataset.N x T x C x H x WDataLoader training usage and backend dependencies.Testing
pytest -q pypaimon/tests/contiguous_window_dataset_test.py pypaimon/tests/multimodal_video_test.py(54 passed)pytest -q pypaimon/tests/multimodal_lerobot_test.py -k 'video_decoder or pyav or indexed'(7 passed)flake8 --config=./dev/cfg.inion changed Python files