gh-148660: Fix use-after-free in OrderedDict.copy() on re-entrant mutation#151531
Open
harjothkhara wants to merge 1 commit into
Open
gh-148660: Fix use-after-free in OrderedDict.copy() on re-entrant mutation#151531harjothkhara wants to merge 1 commit into
harjothkhara wants to merge 1 commit into
Conversation
454e21a to
5576393
Compare
…nt mutation Building the copy iterated od's node list while running arbitrary Python (a key's __eq__/__hash__, a subclass' __getitem__/__setitem__, or a value's __del__) that could clear od and free the nodes being walked, causing a use-after-free. Snapshot od_state and raise RuntimeError if it changes during the copy, as is already done for OrderedDict equality (pythongh-119004).
5576393 to
ea89579
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OrderedDict.copy()walked the source dict's internal linked list while building the new dict. Each step runs arbitrary Python — a key's__eq__/__hash__, a subclass's__getitem__/__setitem__, or a value's__del__— which can callod.clear()(or otherwise mutateod) and free the very nodes being iterated, so the nextnode->nextis a use-after-free (segfault; ASan report in the issue).Following the existing guard for
OrderedDict.__eq__(gh-119004), the copy now snapshotsod_stateand raisesRuntimeError("OrderedDict mutated during iteration")if the dict is mutated mid-copy, before any freed node is dereferenced. Keys are pinned withPy_NewRefacross each re-entrant call.Behavior change:
od.copy()now raisesRuntimeError(instead of crashing/corrupting) when re-entrant code mutatesodduring the copy — the same trade-offOrderedDict.__eq__already makes. A genuine exception raised by the re-entrant call is propagated unchanged.Verification
./configure --with-pydebug && make -j8(clang, macOS arm64);odictobject.ccompiles clean under-Wall -Wextra../python -m test test_ordered_dict→ SUCCESS (309 tests; the new tests run against both the exactOrderedDictand a subclass)../python -m test test_ordered_dict -R 3:3 -m '*issue148660*'→ SUCCESS (no reference leaks)../python -m test test_collections→ SUCCESS.Real behavior proof
Behavior addressed: segfault / use-after-free in
OrderedDict.copy()when a key's__eq__/__hash__, a subclass__getitem__/__setitem__, or a value's__del__mutates the dict during the copy.Real environment tested: CPython
main, built--with-pydebugon macOS arm64 (clang), default (GIL) build.Exact steps or command run after this patch (
poc.pyis the issue reproducer):Evidence after fix:
All four reproducers (
__eq__, subclass__getitem__, subclass__setitem__, value__del__) raise the sameRuntimeErrorinstead of crashing.Observed result after fix: no crash;
copy()raises a catchableRuntimeError; normal/subclass/empty copies are unchanged.What was not tested: the free-threaded (
--disable-gil) build was not exercised at runtime — reasoned about viacopy()'s@critical_section(od)(cross-thread mutation is excluded; same-thread reentrancy is what's guarded). ASan was not run locally; the issue reporter's ASan trace plus a--with-pydebug+faulthandlersegfault establish the crash.This is a crash fix and may be a backport candidate for the active maintenance branches (as gh-119004 was); I'll leave that decision to a maintainer.