macports-base icon indicating copy to clipboard operation
macports-base copied to clipboard

Set SOURCE_DATE_EPOCH

Open harens opened this issue 2 years ago • 4 comments

Closes: https://trac.macports.org/ticket/59672

This PR allows base to define the SOURCE_DATE_EPOCH environment variable to assist with reproducible builds. The value can be manually set in a portfile via the source_date_epoch variable.

If the variable isn't assigned, its value is set to the greatest Unix timestamp of the files present in the source code (credit to Ryan for implementation details).

harens avatar Jun 29 '22 15:06 harens

Just to show how powerful SOURCE_DATE_EPOCH is, I've pushed a commit that standardises the file modification timestamps produced by MacPorts archives.

I tested this with a port I maintain, ioping. I ran sudo port -ds archive ioping twice, and then compared the archives using diffoscope.

Before this PR: File timestamps are different, so the archives are as well.

$ gsha256sum ioping-1.3_0.darwin_21.arm64\ 2.tbz2
6135d8dff9c87a3074fbf1dbaeea13db4ab0ef7774bcd5072e1c5196246d7396  ioping-1.3_0.darwin_21.arm64 2.tbz2
$ gsha256sum ioping-1.3_0.darwin_21.arm64.tbz2
6958d0826cc9c0d2775f3d9f9c20372f2c2fa5369caa44f80b7fbc4746504743  ioping-1.3_0.darwin_21.arm64.tbz2

Afterwards: The archives are completely identical.

$ gsha256sum ioping-1.3_0.darwin_21.arm64\ 2.tbz2
2406a76abc668c91ef9ec066ab9735dbee55ceba615ea6ec15aff6537d6d71ce  ioping-1.3_0.darwin_21.arm64 2.tbz2
$ gsha256sum ioping-1.3_0.darwin_21.arm64.tbz2
2406a76abc668c91ef9ec066ab9735dbee55ceba615ea6ec15aff6537d6d71ce  ioping-1.3_0.darwin_21.arm64.tbz2

Below is the diffoscope output showing how the timestamps in the original archives are different:

@@ -1,14 +1,14 @@
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./
--rw-r--r--   0 root         (0) wheel        (0)      287 2023-02-25 12:13:31.000000 ./+STATE
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./
+-rw-r--r--   0 root         (0) wheel        (0)      287 2023-02-25 12:13:03.000000 ./+STATE
 -rw-r--r--   0 root         (0) wheel        (0)     1039 2022-09-04 11:18:25.000000 ./+PORTFILE
--rw-r--r--   0 root         (0) wheel        (0)      405 2023-02-25 12:13:31.000000 ./+CONTENTS
--rw-r--r--   0 root         (0) wheel        (0)      152 2023-02-25 12:13:31.000000 ./+DESC
--rw-r--r--   0 root         (0) wheel        (0)       39 2023-02-25 12:13:31.000000 ./+COMMENT
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/bin/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/share/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/share/man/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/share/man/man1/
--r--r--r--   0 root         (0) wheel        (0)     3603 2023-02-25 12:13:31.000000 ./opt/local/share/man/man1/ioping.1.gz
--rwxr-xr-x   0 root         (0) wheel        (0)    73539 2023-02-25 12:13:31.000000 ./opt/local/bin/ioping
+-rw-r--r--   0 root         (0) wheel        (0)      405 2023-02-25 12:13:03.000000 ./+CONTENTS
+-rw-r--r--   0 root         (0) wheel        (0)      152 2023-02-25 12:13:03.000000 ./+DESC
+-rw-r--r--   0 root         (0) wheel        (0)       39 2023-02-25 12:13:03.000000 ./+COMMENT
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/bin/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/share/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/share/man/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/share/man/man1/
+-r--r--r--   0 root         (0) wheel        (0)     3603 2023-02-25 12:13:03.000000 ./opt/local/share/man/man1/ioping.1.gz
+-rwxr-xr-x   0 root         (0) wheel        (0)    73539 2023-02-25 12:13:03.000000 ./opt/local/bin/ioping

harens avatar Jun 29 '22 17:06 harens

~~Seem to be a lot of "failed to extract" errors:~~

error: failed to extract dependencies-b: /private/tmp/macports-tests/opt/local/var/macports/build/_users_runner_work_macports-base_macports-base_tests_test_dependencies-b/dependencies-b/work/dependencies-b-1: no such file or directory
...
Error: Failed to extract envvariables: /private/tmp/macports-tests/opt/local/var/macports/build/_Users_runner_work_macports-base_macports-base_tests_test_envvariables/envvariables/work/envvariables-1: no such file or directory
...
etc.

~~Any help debugging appreciated.~~

EDIT: Fix incoming.

harens avatar Feb 08 '23 19:02 harens

Took me a minute of reading the code, but I see now that you implemented:

  1. setting SOURCE_DATE_EPOCH during all phases, and
  2. changing the timestamp of all files in the archive to that value

In the ticket I was only proposing doing (1) and I don't know what the consequences are of doing (2).

ryandesign avatar May 26 '23 06:05 ryandesign

Took me a minute of reading the code, but I see now that you implemented:

1. setting `SOURCE_DATE_EPOCH` during all phases, and

2. changing the timestamp of all files in the archive to that value

In the ticket I was only proposing doing (1) and I don't know what the consequences are of doing (2).

Thank you for the review Ryan, you make a very valid point. I think the easiest thing for now is to simply add SOURCE_DATE_EPOCH to base as was discussed in the original ticket. Standardising the timestamps can be discussed at a later date.

harens avatar Jun 27 '23 14:06 harens