bpo-34093: Stablize FLAG_REF usage (two-pass version)

Doc/library/marshal.rst

@@ -54,7 +54,7 @@ bytes-like objects.
 The module defines these functions:
 
 
-.. function:: dump(value, file[, version])
+.. function:: dump(value, file[, version [, stable]])
 
    Write the value on the open file.  The value must be a supported type.  The
    file must be a writeable :term:`binary file`.
@@ -66,6 +66,13 @@ The module defines these functions:
    The *version* argument indicates the data format that ``dump`` should use
    (see below).
 
+   The *stable* argument makes generated data more stable as possible.
+   It guarantees ``dump(value1, 4, True) == dump(value2, 4, True)``
+   for ``value1 is value2``, but not for ``value1 == value2``.
+
+   .. versionadded:: 3.8
+      *stable* option is added.
+
 
 .. function:: load(file)
 
@@ -80,7 +87,7 @@ The module defines these functions:
       :func:`load` will substitute ``None`` for the unmarshallable type.
 
 
-.. function:: dumps(value[, version])
+.. function:: dumps(value[, version [, stable]])
 
    Return the bytes object that would be written to a file by ``dump(value, file)``.  The
    value must be a supported type.  Raise a :exc:`ValueError` exception if value
@@ -89,6 +96,13 @@ The module defines these functions:
    The *version* argument indicates the data format that ``dumps`` should use
    (see below).
 
+   The *stable* argument makes generated data more stable as possible.
+   It guarantees ``dump(value1, 4, True) == dump(value2, 4, True)``
+   for ``value1 is value2``, but not for ``value1 == value2``.
+
+   .. versionadded:: 3.8
+      *stable* option is added.
+
 
 .. function:: loads(bytes)
 

Lib/test/test_marshal.py

@@ -324,6 +324,25 @@ def test_eof(self):
         for i in range(len(data)):
             self.assertRaises(EOFError, marshal.loads, data[0: i])
 
+    def test_stable_refs(self):
+        """FLAG_REF must be used regardless refcnt"""
+        x = 0x42
+        y = (x,)
+        z = [y, y]
+        dummy = x  # refcnt of x must be >1
+
+        # x is used once, FLAG_REF must not be set.
+        data = marshal.dumps(x, 4, True)
+        self.assertEqual(b"i\x42\x00\x00\x00", data)
+
+        data = marshal.dumps(z, 4, True)
+        # y is used twice, but x is used once because y is reused.
+        self.assertEqual(b"[\x02\x00\x00\x00" +     # list(size=2)i\x42\x00\x00\x00", data)
+                         b"\xa9\x01" +              # small tuple(size=1) | FLAG_REF
+                         b"i\x42\x00\x00\x00" +     # int(42)
+                         b"r\x00\x00\x00\x00",      # ref(0)
+                         data)
+
 LARGE_SIZE = 2**31
 pointer_size = 8 if sys.maxsize > 0xFFFFFFFF else 4