grass

grass copied to clipboard

Allow to override build date with SOURCE_DATE_EPOCH because in builds from a release tarball, there is no git commit to take a date from, so the current date was inserted there.

See https://reproducible-builds.org/ for why this matters and https://reproducible-builds.org/specs/source-date-epoch/ for the definition of this variable.

This patch was done while working on reproducible builds for openSUSE.

note: I only tested the build and only on 8.2.1

I'll take a closer look later, but I'm really favorable on the concept. Thanks!

When GRASS GIS is compiled from the release tarball source code (without Git, and without official patch https://github.com/OSGeo/grass/releases/download/8.3.1/core_modules_with_last_commit.patch) getting date is provide with this function get_default_git_log() (line 218) :

https://github.com/OSGeo/grass/blob/aaecfc8957ab906785b3231da2aed7d0861ce71a/utils/mkhtml.py#L407-L409

https://github.com/OSGeo/grass/blob/aaecfc8957ab906785b3231da2aed7d0861ce71a/utils/mkhtml.py#L259-L261

https://github.com/OSGeo/grass/blob/aaecfc8957ab906785b3231da2aed7d0861ce71a/utils/mkhtml.py#L205-L221

Compilation was tested with GRASS GIS official release tarball source code 8.3.1 version under Gentoo GNU/Linux distribution.

r.surf.rst module man page example:

Ok I see patch is for reproducible builds for openSUSE distribution.

Small note:

You can use official patch core_modules_with_last_commit.patch or simple core_modules_with_last_commit.json file for getting commits for GRASS GIS core modules for compilation under openSUSE distribution.

Patch/JSON file is generated for every official release, you can find them under release URL https://github.com/OSGeo/grass/releases page under Assets section.

Unfortunately we are still on 8.2.1 atm and there is no https://github.com/OSGeo/grass/releases/download/8.2.1/core_modules_with_last_commit.json / patch - is there a way to only take my code-path for when that file is missing?

Unfortunately we are still on 8.2.1 atm and there is no https://github.com/OSGeo/grass/releases/download/8.2.1/core_modules_with_last_commit.json / patch - is there a way to only take my code-path for when that file is missing?

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

Doesn't return this line code datetime.fromtimestamp(os.path.getmtime(src_dir)) (code return the time of last modification of path of source code dir) valid date time during reproducible build process from release tarball for openSUSE distro?

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

@tmszi Could that be backported to 8.2?

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

@tmszi Could that be backported to 8.2?

I think so, but I will have to look into the source code (and prepare backport PR) to give you the exact answer.

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

@tmszi Could that be backported to 8.2?

The source code that is needed to automatically generate core modules with the last commit JSON file and patch file is included in the realeasebranch_8_2 branch.

I'm wonder why these files is not automatically created for 8.2.x realease. GitHub workflow config files looks good to me.

https://github.com/OSGeo/grass/blob/0bd5c8d434532a31f890a307ce4c08273b8d70eb/.github/workflows/create_release_draft.yml#L34-L38

https://github.com/OSGeo/grass/blob/0bd5c8d434532a31f890a307ce4c08273b8d70eb/.github/workflows/create_release_draft.yml#L62-L71

https://github.com/OSGeo/grass/blob/0bd5c8d434532a31f890a307ce4c08273b8d70eb/.github/workflows/create_release_draft.yml#L95-L104

@bmwiedemann It is possible generate JSON/patch file manually for 8.2.1 version:

1. git clone -b relasebranch_8_2 --single-branch https://github.com/OSGeo/grass.git && cd grass/
2. git checkout tags/8.2.1
3. python utils/generate_last_commit_file.py .
4. file core_modules_with_last_commit.json
5. git add core_modules_with_last_commit.json
6. git diff --cached > core_modules_with_last_commit.patch
7. file core_modules_with_last_commit.patch

Wouldn't it be better to – in the process of creating the tarball – create and add a simple textfile containing the git commit? Configure then can try get info from git or if failing so retrieve the same info from the file (if existing).

Wouldn't it be better to – in the process of creating the tarball – create and add a simple textfile containing the git commit? Configure then can try get info from git or if failing so retrieve the same info from the file (if existing).

Must be better than:

https://github.com/OSGeo/grass/blob/25bc70d0f56d4fe6e9b426803459aad499e41251/configure.ac#L157-L176

…or did I miss something?

Wouldn't it be better to – in the process of creating the tarball – create and add a simple textfile containing the git commit? Configure then can try get info from git or if failing so retrieve the same info from the file (if existing).

Yes it is improvement, but core_modules_with_last_commit patch/JSON file is not required file by default.

Actually all states are handled correctly during compilation process:

1. Git commits/date are getting from Git source code if exists
2. Git source code doesn't exists (tarball) Git commits/date are getting from JSON file if exists
3. Git source code or JSON file doesn't exists Git commit info will be empty and date will be time of last modification of source code path (line 218)

Info from the man page will looks like: Accessed: Sunday Feb 18 10:58:31 2024

https://github.com/OSGeo/grass/blob/aaecfc8957ab906785b3231da2aed7d0861ce71a/utils/mkhtml.py#L216-L221

In practice, os.path.getmtime(src_dir) results in the current time being used, because we do some sed calls

Yes it is improvement, but core_modules_with_last_commit patch/JSON file is not required file by default.

The generated patch/json file is needed in a tarball release, but should be included in the tarball and used by configure. This way it results in a consistent manner.

#3435 attempt to address this for tarball releases with GRASS_HEADERS_GIT_HASH and GRASS_HEADERS_GIT_DATE, without the need for introducing new environmental variables and even further complicate the code. Next step will be to update configure and/or utils/mkhtml.py to read, if needed, from distributed json file.

Next step will be to update configure and/or utils/mkhtml.py to read, if needed, from distributed json file.

@tmszi I have not yet studied the code, but if we package the json file in the tarball, will it then automatically be used? At least that is my understanding after re-reading your comment.

Next step will be to update configure and/or utils/mkhtml.py to read, if needed, from distributed json file.

@tmszi I have not yet studied the code, but if we package the json file in the tarball, will it then automatically be used? At least that is my understanding after re-reading your comment.

Yes it will be.

core_modules_with_last_commit.json file structure:

{
    "g.version": {
        "commit": "f146b2078dadabb79722b8e6b9323b8ae535f40f",
        "date": "2022-12-13T16:22:24-03:00"
    },
    "g.filename": {
        "commit": "547ff44e6aecfb4c9cbf6a4717fc14e521bec0be",
        "date": "2022-02-03T11:10:06+01:00"
    },
} 

So now that #3435 is done, what is the status of this PR?

With #3435 and #3437, resulting in Git related data included in tarball too, consistent builds are enabled with/without Git and also across platforms. This will be available with GRASS GIS 8.4. @bmwiedemann Thank you for bringing up this issue. Please don't hesitate to report an issue if the fixes for 8.4 will not create reproducible builds (for Git derived data).