Skip to content

fix ref bug with iter, views and istr#1311

Open
Vizonex wants to merge 17 commits intoaio-libs:masterfrom
Vizonex:iterator-ref-leak
Open

fix ref bug with iter, views and istr#1311
Vizonex wants to merge 17 commits intoaio-libs:masterfrom
Vizonex:iterator-ref-leak

Conversation

@Vizonex
Copy link
Copy Markdown
Member

@Vizonex Vizonex commented Mar 28, 2026

What do these changes do?

This is based off previous things that I have referenced in #1310 and #1306. The other bugs listed seem to like much harder puzzles to solve than anything else so I'm tackling the easy fixable ones first.

Reproducer

"""Reproducer: multidict iterator/view/istr types leak type references.

iter.h:134 — multidict_iter_dealloc: no Py_DECREF(Py_TYPE(self))
views.h:30 — multidict_view_dealloc: no Py_DECREF(Py_TYPE(self))
istr.h:22 — istr_dealloc: no Py_DECREF(Py_TYPE(self))
traverse functions also missing Py_VISIT(Py_TYPE(self)).

Tested: multidict 6.7.1, Python 3.14.
"""
from multidict import MultiDict, istr
import sys

md = MultiDict([("a", "1"), ("b", "2")])

# Test iterator type leak
it = iter(md.keys())
iter_type = type(it)
del it
baseline = sys.getrefcount(iter_type)
for i in range(1000):
    it = iter(md.keys())
    list(it)
    del it
after = sys.getrefcount(iter_type)
print(f"KeysIter type: baseline={baseline}, after={after}, leaked={after-baseline}")

# Test view type leak
view_type = type(md.keys())
baseline = sys.getrefcount(view_type)
for i in range(1000):
    v = md.keys()
    del v
after = sys.getrefcount(view_type)
print(f"KeysView type: baseline={baseline}, after={after}, leaked={after-baseline}")

# Test istr type leak
baseline = sys.getrefcount(istr)
for i in range(1000):
    s = istr("hello")
    del s
after = sys.getrefcount(istr)
print(f"istr type:     baseline={baseline}, after={after}, leaked={after-baseline}")

# Expected: 0 leaked for all
# Actual: exactly 1000 leaked for each — one per object lifecycle

Are there changes in behavior for the user?

Related issue number

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes

@Vizonex Vizonex requested a review from asvetlov as a code owner March 28, 2026 19:22
@Vizonex Vizonex requested a review from webknjaz as a code owner March 28, 2026 19:25
@psf-chronographer psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Mar 28, 2026
@bdraco
Copy link
Copy Markdown
Member

bdraco commented Mar 29, 2026

Would you please add a test for this one as well ?

@Vizonex
Copy link
Copy Markdown
Member Author

Vizonex commented Mar 30, 2026

Would you please add a test for this one as well ?

Just did. :)

@Vizonex
Copy link
Copy Markdown
Member Author

Vizonex commented Mar 30, 2026

Seems this implementation likes to segfault :( I'll probably go back and replan a few things tomorrow when I have 2 days off from my job.

Vizonex added 2 commits March 31, 2026 12:45
…ypes after finishing and use a tp_free slot to help with that.
@Vizonex
Copy link
Copy Markdown
Member Author

Vizonex commented Mar 31, 2026

@bdraco I seem to have just fixed the problem. Turns out using a Py_tp_free can be very useful. I think what occurred was that the rewrite was modeled after the python standard library a little bit too deeply. I decided it might be smart to try another implementation such as freeing the type afterwards.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Mar 31, 2026

Merging this PR will not alter performance

✅ 245 untouched benchmarks


Comparing Vizonex:iterator-ref-leak (c06e3a0) with master (f74b0b8)

Open in CodSpeed

@Vizonex
Copy link
Copy Markdown
Member Author

Vizonex commented Mar 31, 2026

The only bad news is that freethreaded is still bugged.

… instead and recast istr directly instead of using _PyObject_CAST
@Vizonex
Copy link
Copy Markdown
Member Author

Vizonex commented Mar 31, 2026

@bdraco It should also be mentioned that code-coverage is somehow failing in the isolated directory. I've looked at them and most files seem to act that way at around 33.33% or similar results. I wonder if this coverage is inaccurate or should be fixed?

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 31, 2026

Codecov Report

❌ Patch coverage is 11.11111% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.94%. Comparing base (f74b0b8) to head (c06e3a0).

Files with missing lines Patch % Lines
tests/isolated/multidict_type_leak.py 11.11% 23 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (11.11%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (96.08%) is below the target coverage (100.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1311      +/-   ##
==========================================
- Coverage   97.60%   96.94%   -0.66%     
==========================================
  Files          26       27       +1     
  Lines        3513     3540      +27     
  Branches      253      257       +4     
==========================================
+ Hits         3429     3432       +3     
- Misses         77      100      +23     
- Partials        7        8       +1     
Flag Coverage Δ
CI-GHA 96.94% <11.11%> (-0.66%) ⬇️
pytest 96.94% <11.11%> (-0.66%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

view_type = type(md.keys())
baseline = sys.getrefcount(view_type)
for i in range(1000):
v = md.keys()

Check notice

Code scanning / CodeQL

Unused global variable Note test

The global variable 'v' is not used.

baseline = sys.getrefcount(istr)
for i in range(1000):
s = istr("hello")

Check notice

Code scanning / CodeQL

Unused global variable Note test

The global variable 's' is not used.
@Vizonex Vizonex mentioned this pull request Apr 3, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants