From f6e54c2473a56df84ecc7e68876c3eeb48a130bd Mon Sep 17 00:00:00 2001 From: Julien LENORMAND Date: Sat, 18 Jul 2026 10:08:02 +0200 Subject: [PATCH 1/4] docs: warn about results asymmetry for difflib junk --- Doc/library/difflib.rst | 77 +++++++++++++++++-- ...-07-17-22-33-43.gh-issue-118150.m7iFdP.rst | 3 + 2 files changed, 73 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 25edb40e35a630a..225c3f5998acfaf 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -40,6 +40,16 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. complicated way on how many elements the sequences have in common; best case time is linear. + **User-defined junk predicate**: :class:`SequenceMatcher` accepts an *isjunk* + parameter to let users determine which lines should be ignored while computing + the diff. + + .. note:: + + This predicate only ever inspects the second sequence for junk. That has + an unexpected consequence: swapping the inputs can produce a different + diff than simply reversing the output, sometimes very large. + **Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that automatically treats certain sequence items as junk. The heuristic counts how many times each individual item appears in the sequence. If an item's duplicates (after @@ -48,6 +58,12 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. the purpose of sequence matching. This heuristic can be turned off by setting the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`. + .. note:: + + This heuristic only ever inspects the second sequence for junk. That has + an unexpected consequence: swapping the inputs can produce a different + diff than simply reversing the output, sometimes very large. + .. versionchanged:: 3.2 Added the *autojunk* parameter. @@ -189,6 +205,11 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. + .. note:: + + This uses a :class:`SequenceMatcher` internally, so swapping *a* and + *b* can produce different results. + .. function:: get_close_matches(word, possibilities, n=3, cutoff=0.6) @@ -250,6 +271,11 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. + tree + emu + .. note:: + + This uses a :class:`SequenceMatcher` internally, so swapping *a* and + *b* can produce different results. + .. function:: restore(sequence, which) @@ -319,6 +345,11 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. + .. note:: + + This uses a :class:`SequenceMatcher` internally, so swapping *a* and + *b* can produce different results. + .. versionchanged:: 3.15 Added the *color* parameter. @@ -397,6 +428,18 @@ The :class:`SequenceMatcher` class has this constructor: of positions where they occur. All three are reset whenever *b* is reset with :meth:`set_seqs` or :meth:`set_seq2`. + .. note:: + + Both *isjunk* and the *autojunk* heuristic only consider *b* for filtering + out items. Consequently, swapping *a* and *b* can change which items are treated + as junk, which can in turn change the matching blocks found by + :meth:`get_matching_blocks` and :meth:`get_opcodes`, the value returned by + :meth:`ratio`, and the diffs produced by :func:`unified_diff`, + :func:`context_diff`, and :func:`ndiff`, all of which build a + :class:`SequenceMatcher` internally. + A custom *isjunk* has this effect at any sequence length, and the automatic + *autojunk* heuristic has the same effect for sequences of 200 or more items. + .. versionadded:: 3.2 The *bjunk* and *bpopular* attributes. @@ -486,6 +529,21 @@ The :class:`SequenceMatcher` class has this constructor: >>> s.get_matching_blocks() [Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)] + .. note:: + + Caution: the result may depend on the order of the arguments. For + instance:: + + >>> SequenceMatcher(None, 'tide', 'diet').get_matching_blocks() + [Match(a=0, b=3, size=1), Match(a=4, b=4, size=0)] + >>> SequenceMatcher(None, 'diet', 'tide').get_matching_blocks() + [Match(a=0, b=2, size=1), Match(a=2, b=3, size=1), Match(a=4, b=4, size=0)] + + This comes from the matching algorithm itself. Another cause for order + causing different outputs, sometimes wildly, is the junk filtering + heuristic (a custom *isjunk* at any sequence length, or the + automatic *autojunk* heuristic for sequences of 200 or more + items). See the :class:`SequenceMatcher` for details. .. method:: get_opcodes() @@ -527,6 +585,11 @@ The :class:`SequenceMatcher` class has this constructor: equal a[4:6] --> b[3:5] 'cd' --> 'cd' insert a[6:6] --> b[5:6] '' --> 'f' + .. note:: + + This is derived from :meth:`get_matching_blocks`, and so shares + its order-dependence; see the note there. + .. method:: get_grouped_opcodes(n=3) @@ -538,6 +601,11 @@ The :class:`SequenceMatcher` class has this constructor: The groups are returned in the same format as :meth:`get_opcodes`. + .. note:: + + This is derived from :meth:`get_matching_blocks`, and so shares + its order-dependence; see the note there. + .. method:: ratio() @@ -555,13 +623,8 @@ The :class:`SequenceMatcher` class has this constructor: .. note:: - Caution: The result of a :meth:`ratio` call may depend on the order of - the arguments. For instance:: - - >>> SequenceMatcher(None, 'tide', 'diet').ratio() - 0.25 - >>> SequenceMatcher(None, 'diet', 'tide').ratio() - 0.5 + This is derived from :meth:`get_matching_blocks`, and so shares + its order-dependence; see the note there. .. method:: quick_ratio() diff --git a/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst b/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst new file mode 100644 index 000000000000000..c7ef2296204f60d --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst @@ -0,0 +1,3 @@ +Clarify in the :mod:`difflib` documentation that both a custom *isjunk* +predicate and the *autojunk* heuristic of :class:`~difflib.SequenceMatcher` +can treat ``(a, b)`` and ``(b, a)`` differently. From 39e47c3506df79572b63bcf8ceaa5259da2a75fe Mon Sep 17 00:00:00 2001 From: Julien LENORMAND Date: Sat, 18 Jul 2026 17:13:49 +0200 Subject: [PATCH 2/4] merge and reword the whole junk mechanism --- Doc/library/difflib.rst | 35 ++++++++++++----------------------- 1 file changed, 12 insertions(+), 23 deletions(-) diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 225c3f5998acfaf..728370ae3694cdc 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -40,29 +40,18 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. complicated way on how many elements the sequences have in common; best case time is linear. - **User-defined junk predicate**: :class:`SequenceMatcher` accepts an *isjunk* - parameter to let users determine which lines should be ignored while computing - the diff. - - .. note:: - - This predicate only ever inspects the second sequence for junk. That has - an unexpected consequence: swapping the inputs can produce a different - diff than simply reversing the output, sometimes very large. - - **Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that - automatically treats certain sequence items as junk. The heuristic counts how many - times each individual item appears in the sequence. If an item's duplicates (after - the first one) account for more than 1% of the sequence and the sequence is at least - 200 items long, this item is marked as "popular" and is treated as junk for - the purpose of sequence matching. This heuristic can be turned off by setting - the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`. - - .. note:: - - This heuristic only ever inspects the second sequence for junk. That has - an unexpected consequence: swapping the inputs can produce a different - diff than simply reversing the output, sometimes very large. + **Junk**: :class:`SequenceMatcher` accepts an ``isjunk`` predicate and an + ``autojunk`` flag. Items that are considered as junk will not be considered + to find similar content blocks. This can produce better results for humans + (typically breaking on whitespace) and faster (because it reduces the number + of possible combinations). But it can also cause pathological cases where + too many items considered junk cause an unexpectedly large (but correct) + diff result. + You should consider tuning them or turning them off depending on your data. + Moreover, only the second sequence is inspected for junk. This causes the diff + output to not be symmetrical. + When ``autojunk=True``, it will consider as junk the items that account for more + than 1% of the sequence, if it is at least 200 items long. .. versionchanged:: 3.2 Added the *autojunk* parameter. From 1d9605ea67cfad182e9d91d293cc1237c3a2ae8f Mon Sep 17 00:00:00 2001 From: Julien LENORMAND Date: Sat, 18 Jul 2026 17:30:54 +0200 Subject: [PATCH 3/4] remove all notes, keep only junk explanation at the top --- Doc/library/difflib.rst | 57 ----------------------------------------- 1 file changed, 57 deletions(-) diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 728370ae3694cdc..0cebd355ff5e250 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -194,11 +194,6 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. - .. note:: - - This uses a :class:`SequenceMatcher` internally, so swapping *a* and - *b* can produce different results. - .. function:: get_close_matches(word, possibilities, n=3, cutoff=0.6) @@ -260,11 +255,6 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. + tree + emu - .. note:: - - This uses a :class:`SequenceMatcher` internally, so swapping *a* and - *b* can produce different results. - .. function:: restore(sequence, which) @@ -334,11 +324,6 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. - .. note:: - - This uses a :class:`SequenceMatcher` internally, so swapping *a* and - *b* can produce different results. - .. versionchanged:: 3.15 Added the *color* parameter. @@ -417,18 +402,6 @@ The :class:`SequenceMatcher` class has this constructor: of positions where they occur. All three are reset whenever *b* is reset with :meth:`set_seqs` or :meth:`set_seq2`. - .. note:: - - Both *isjunk* and the *autojunk* heuristic only consider *b* for filtering - out items. Consequently, swapping *a* and *b* can change which items are treated - as junk, which can in turn change the matching blocks found by - :meth:`get_matching_blocks` and :meth:`get_opcodes`, the value returned by - :meth:`ratio`, and the diffs produced by :func:`unified_diff`, - :func:`context_diff`, and :func:`ndiff`, all of which build a - :class:`SequenceMatcher` internally. - A custom *isjunk* has this effect at any sequence length, and the automatic - *autojunk* heuristic has the same effect for sequences of 200 or more items. - .. versionadded:: 3.2 The *bjunk* and *bpopular* attributes. @@ -518,21 +491,6 @@ The :class:`SequenceMatcher` class has this constructor: >>> s.get_matching_blocks() [Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)] - .. note:: - - Caution: the result may depend on the order of the arguments. For - instance:: - - >>> SequenceMatcher(None, 'tide', 'diet').get_matching_blocks() - [Match(a=0, b=3, size=1), Match(a=4, b=4, size=0)] - >>> SequenceMatcher(None, 'diet', 'tide').get_matching_blocks() - [Match(a=0, b=2, size=1), Match(a=2, b=3, size=1), Match(a=4, b=4, size=0)] - - This comes from the matching algorithm itself. Another cause for order - causing different outputs, sometimes wildly, is the junk filtering - heuristic (a custom *isjunk* at any sequence length, or the - automatic *autojunk* heuristic for sequences of 200 or more - items). See the :class:`SequenceMatcher` for details. .. method:: get_opcodes() @@ -574,11 +532,6 @@ The :class:`SequenceMatcher` class has this constructor: equal a[4:6] --> b[3:5] 'cd' --> 'cd' insert a[6:6] --> b[5:6] '' --> 'f' - .. note:: - - This is derived from :meth:`get_matching_blocks`, and so shares - its order-dependence; see the note there. - .. method:: get_grouped_opcodes(n=3) @@ -590,11 +543,6 @@ The :class:`SequenceMatcher` class has this constructor: The groups are returned in the same format as :meth:`get_opcodes`. - .. note:: - - This is derived from :meth:`get_matching_blocks`, and so shares - its order-dependence; see the note there. - .. method:: ratio() @@ -610,11 +558,6 @@ The :class:`SequenceMatcher` class has this constructor: to try :meth:`quick_ratio` or :meth:`real_quick_ratio` first to get an upper bound. - .. note:: - - This is derived from :meth:`get_matching_blocks`, and so shares - its order-dependence; see the note there. - .. method:: quick_ratio() From 079374086b0e6a7df258d6b113388d27f410ef22 Mon Sep 17 00:00:00 2001 From: Julien LENORMAND Date: Sat, 18 Jul 2026 17:36:52 +0200 Subject: [PATCH 4/4] update NEWS to the current content of the PR --- .../2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst b/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst index c7ef2296204f60d..afa6007a46e1745 100644 --- a/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst +++ b/Misc/NEWS.d/next/Documentation/2026-07-17-22-33-43.gh-issue-118150.m7iFdP.rst @@ -1,3 +1,2 @@ -Clarify in the :mod:`difflib` documentation that both a custom *isjunk* -predicate and the *autojunk* heuristic of :class:`~difflib.SequenceMatcher` -can treat ``(a, b)`` and ``(b, a)`` differently. +Clarify in the :mod:`difflib` documentation what *junk* actually does, its +drawbacks, and how to control it.