bpo-29708: Setting SOURCE_DATE_EPOCH forces hash-based .pyc files (GH-5200)

To support reproducible builds, the setting of of SOURCE_DATE_EPOCH triggers the py_compile module -- and by extension, compileall -- to forcibly compile with hash-based .pyc files. This eliminates the possibility of timestamp-based .pyc files which vary between builds.

Author: bmwiedemann

Doc/library/py_compile.rst

@@ -55,7 +55,9 @@ byte-code cache files in the directory containing the source code.
 
    *invalidation_mode* should be a member of the :class:`PycInvalidationMode`
    enum and controls how the generated ``.pyc`` files are invalidated at
-   runtime.
+   runtime. If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
+   *invalidation_mode* will be forced to
+   :attr:`PycInvalidationMode.CHECKED_HASH`.
 
    .. versionchanged:: 3.2
       Changed default value of *cfile* to be :PEP:`3147`-compliant.  Previous
@@ -71,6 +73,9 @@ byte-code cache files in the directory containing the source code.
 
    .. versionchanged:: 3.7
       The *invalidation_mode* parameter was added as specified in :pep:`552`.
+      If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
+      *invalidation_mode* will be forced to
+      :attr:`PycInvalidationMode.CHECKED_HASH`.
 
 
 .. class:: PycInvalidationMode

Lib/py_compile.py

@@ -112,6 +112,8 @@ def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1,
     the resulting file would be regular and thus not the same type of file as
     it was previously.
     """
+    if os.environ.get('SOURCE_DATE_EPOCH'):
+        invalidation_mode = PycInvalidationMode.CHECKED_HASH
     if cfile is None:
         if optimize >= 0:
             optimization = optimize if optimize >= 1 else ''

Lib/test/test_py_compile.py

@@ -98,6 +98,18 @@ def test_bad_coding(self):
         self.assertFalse(os.path.exists(
             importlib.util.cache_from_source(bad_coding)))
 
+    def test_source_date_epoch(self):
+        testtime = 123456789
+        with support.EnvironmentVarGuard() as env:
+            env["SOURCE_DATE_EPOCH"] = str(testtime)
+            py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+        with open(self.pyc_path, 'rb') as fp:
+            flags = importlib._bootstrap_external._classify_pyc(
+                fp.read(), 'test', {})
+        self.assertEqual(flags, 0b11)
+
     @unittest.skipIf(sys.flags.optimize > 0, 'test does not work with -O')
     def test_double_dot_no_clobber(self):
         # http://bugs.python.org/issue22966

Misc/NEWS.d/next/Build/2018-01-16-08-32-49.bpo-29708.YCaHEx.rst

@@ -0,0 +1,2 @@
+If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
+:mod:`py_compile` will always create hash-based ``.pyc`` files.