Skip to content

Keep julia_to_cpp's size and step arrays alive until cv::Mat has copied them - #102

Merged
yakir12 merged 4 commits into
JuliaImages:masterfrom
yakir12:gc-preserve-mat-sizes
Oct 5, 2026
Merged

yakir12 merged 4 commits into
JuliaImages:masterfrom
yakir12:gc-preserve-mat-sizes

Conversation

@yakir12

@yakir12 yakir12 commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

AI-authored. This PR — the diagnosis, the fix, the test and this description — was written by an AI agent (Claude Code, model Claude Opus 5.5), working for @yakir12, who reviewed it before it was opened.

Summary

julia_to_cpp(::InputArray) builds two small Julia arrays, the Mat's sizes and steps, and passes only their raw pointers to the cv::Mat constructor. Nothing keeps those arrays alive after pointer(...) is taken. A garbage collection triggered by another thread in that window can free and reuse them, and cv::Mat then reads garbage dimensions.

 function julia_to_cpp(img::InputArray)
     ...
-    return CxxMat(2, pointer(ndims_a), cvtype, Ptr{Nothing}(pointer(img)), pointer(steps_a))
+    return GC.@preserve ndims_a steps_a CxxMat(2, pointer(ndims_a), cvtype, Ptr{Nothing}(pointer(img)), pointer(steps_a))

cv::Mat copies sizes and steps into its own storage while it is constructed, so keeping the arrays alive across the constructor call is enough. The pixel data stays borrowed from img, exactly as before.

The race needs ≥ 2 threads. In the wild it showed up in a 32-thread pipeline, as findChessboardCorners throwing:

OpenCV(4.13.0) …/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 65512708266960 bytes

The pipeline was verifying an ordinary, cleanly decoded 1920×1080 frame. Rerunning the identical job succeeded.

Evidence

The new test in test/test_conversions.jl runs julia_to_cpp → cpp_to_julia round-trips on half the threads for 15 s, while the other half allocate and call GC.gc(false). It counts results whose size differs from the input's.

for 15 s on nthreads÷2 tasks, while nthreads÷2 tasks force GC:
  @test size(cpp_to_julia(julia_to_cpp(a))) == size(a)   # counted; assert 0 bad

All runs used Julia 1.13.1 with -t4, the thread count of the multithreads CI job.

  • Before: Test Failed on the first attempt. Most corrupted conversions come back as a Mat with wrong dimensions and no error. Others throw (-215:Assertion failed) s >= 0 in function 'setSize' from CxxMat(...) at mat_conversion.jl:102.
  • After: 6 runs out of 6 pass, about 90 s of hammering in total. test_core_types.jl, test_conversions.jl and test_mat.jl pass (189/189). I did not run the full suite locally, because it needs the opencv_extra artifact.

The test adds about 15 s to the suite. On a single thread it only checks the round-trip, since the race cannot happen there.

Merge Danger

Door: two-way

Blast Radius: small. The first fix only keeps two arrays alive slightly longer. The second affects only non-strided and out-of-order-strided inputs, which now cost a second copy (below).

Second fix: inputs that are copied before conversion

Added after review, at the reviewer's suggestion.

Any input OpenCV cannot borrow gets copied first. That covers a non-strided input (e.g. view(a, :, [1, 3], :)) and one with strides out of order (e.g. a PermutedDimsArray). julia_to_cpp then returned a Mat that borrowed that temporary copy's memory, and nothing kept the copy alive. This bug needs no threads: one collection after the conversion frees the copy under the Mat.

Both copy branches now hand the temporary to cv::copyTo, which copies it into memory OpenCV allocates and refcounts, so the returned Mat owns its pixels:

 julia_to_cpp(img)
   strides(img) throws
-    img = img[:, :, :]                      → falls through, Mat borrows the copy
+    return _owned_cpp_mat(img)
   strides out of order
-    return julia_to_cpp(img[:, :, :])       → Mat borrows the copy
+    return _owned_cpp_mat(img)

+_owned_cpp_mat(img)
+  dense = Array(img)
+  GC.@preserve dense  copyTo(julia_to_cpp(dense), noArray, emptyMat)   → OpenCV-owned Mat

The new testset converts one input of each kind, forces a collection, refills the freed memory with 0xAB, and reads the Mat back.

  • Before: the test fails for both inputs, and the Mat reads back all 0xAB. A standalone reproduction with a 3×1000×800 input segfaults in getindex on the Mat. At 3×20×10 the result is corrupted in 20 of 20 tries for each input kind.
  • After: the test passes, and the reproduction shows 0 of 20 corrupted at both sizes. Round-trips through both copy branches match for all seven supported element types.

With both fixes, the full suite passes with -t4 on Julia 1.13.1: 313/313, opencv_extra artifact included.

Cost: these inputs, which were already copied once, are now copied twice, once into a Julia Array and once into OpenCV's buffer. The wrapper exposes no way to allocate a cv::Mat that Julia then fills, so avoiding the second copy would need a C++ change. If such an input is passed as an output argument (dst), OpenCV now writes into its own copy instead of a freed one. The caller's array was never written in either case.

🤖 Generated with Claude Code

Under concurrent GC the raw pointers handed to the cv::Mat constructor
could outlive the Julia arrays, giving Mats with garbage dimensions
(wrong sizes, setSize assertions, terabyte allocations).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.79%. Comparing base (415d21e) to head (93e6a85).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #102      +/-   ##
==========================================
+ Coverage   91.45%   92.79%   +1.34%     
==========================================
  Files          11       11              
  Lines         234      236       +2     
==========================================
+ Hits          214      219       +5     
+ Misses         20       17       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

On one thread the collector task never yielded, so the converter never
ran and the test spun until CI's 6 h limit. Run the stress part only
when there is a second thread, and yield in both loops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ViralBShah

Copy link
Copy Markdown
Member

LGTM. I would also do the part it left out.

yakir12 and others added 2 commits October 5, 2026 15:33
`julia_to_cpp` copies an input OpenCV cannot borrow (no strides, or strides
out of order) into a temporary dense array, then returned a cv::Mat that
borrowed that temporary's memory. Nothing kept the temporary alive, so a
collection freed it while the Mat still pointed into it.

Both copy branches now hand the temporary to cv::copyTo, which copies it into
memory OpenCV allocates and refcounts, so the returned Mat owns its pixels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The codecov/patch status on this PR has sat at "pending" for more than
five hours, though every test job passed and codecov/project reported.
This empty commit changes no code; it only starts a fresh round of
checks so codecov receives new uploads and can post the patch status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yakir12

yakir12 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed an empty commit (93e6a85) to retrigger CI. The codecov/patch status had been stuck at pending for over five hours, even though every test job passed and codecov/project reported. The commit changes no code. It only starts a fresh round of checks so codecov gets new uploads and can post the patch status.

@yakir12

yakir12 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

phew... I had to manually restart some of the CI jobs. But everything is cleared now. Merging.

@yakir12
yakir12 merged commit 244e22c into JuliaImages:master Oct 5, 2026
21 of 27 checks passed
yakir12 added a commit to yakir12/Fromage.jl that referenced this pull request Oct 6, 2026
…#405)

OpenCV.jl's julia_to_cpp passed raw pointers to Julia-owned size and step
arrays into cv::Mat without GC.@preserve, so a collection on another thread
could hand cv::Mat garbage dimensions. Fromage's threaded verification calls
findChessboardCorners through that conversion; it crashed evri/chunk_008 in
the v0.7.2 re-track. JuliaImages/OpenCV.jl#102 fixed it and is released as
4.10.0. The simulation's test oracle gets the same floor.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants