Skip to content

[python] Support video frames in contiguous window datasets - #10511

Open
YannByron wants to merge 1 commit into
apache:masterfrom
YannByron:codex/cwd-video-frame-field
Open

YannByron wants to merge 1 commit into
apache:masterfrom
YannByron:codex/cwd-video-frame-field

Conversation

@YannByron

Copy link
Copy Markdown
Contributor

Summary

ContiguousWindowDataset currently rejects video-frame-field columns, so callers cannot train directly from video-backed windows. This change keeps video descriptors through row selection and decodes the selected frames only after overlapping window reads have been coalesced.

Changes

  • Decode video-frame-field columns in ContiguousWindowDataset with the existing PyAV and TorchCodec decoder path.
  • Group rows by physical video and decode each requested frame once per batched read while preserving window order.
  • Keep ordinary BLOB body fetching unchanged and route indexed video range I/O through the existing blob_parallelism setting.
  • Add video_backend selection to ScanQuery.to_contiguous_window_dataset.
  • Document direct N x T x C x H x W DataLoader training usage and backend dependencies.

Testing

  • pytest -q pypaimon/tests/contiguous_window_dataset_test.py pypaimon/tests/multimodal_video_test.py (54 passed)
  • pytest -q pypaimon/tests/multimodal_lerobot_test.py -k 'video_decoder or pyav or indexed' (7 passed)
  • Real MP4 encode/decode through a two-worker PyTorch DataLoader
  • flake8 --config=./dev/cfg.ini on changed Python files
  • Python license header check (960 files)

Preserve video frame descriptors through row selection, coalesce frame
decoding across a batch, and expose backend selection for training windows.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-6-astra
Co-Authored-By: Codex <noreply@ai-tool.com>
Co-Authored-By: Codex <noreply@openai.com>
AI-Contributed/Feature: 181/181
AI-Contributed/UT: 103/103

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one reproducible memory issue in the video decoding path; details inline.

def _decode_video_frames(decoder, frame_indices, unused_rows):
get_frames_at = getattr(decoder, "get_frames_at", None)
if get_frames_at is not None:
return get_frames_at(indices=frame_indices).data

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Give each decoded frame independent storage

get_frames_at(...).data is a batched Tensor, so the collator assigns views of its storage to individual rows. _sample() then calls copy.deepcopy() separately for each requested frame (lines 303–305), which copies the entire backing batch storage for every view. With no column_transforms, a real PyAV read of 32 three-frame windows (34 decoded rows) returned 96 independent storages of 4,896 bytes each, although each 8×6 RGB frame contains only 144 bytes: 470,016 retained bytes instead of 13,824 (34×). torch.stack releases those oversized copies after each transform, but still incurs the full batch copies. This substantially increases memory and copying costs for video-training batches. Please clone each frame before passing it to the collator, or use Tensor.clone() instead of deepcopy in _sample(), and add a regression check for the returned frames' storage size. Per-frame cloning reduced this reproduction to 1× storage with identical frame values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants