Support copying/linking files in manifest

[tknapstad]

Use case

I wish to add files to the top level west workspace, for instance a README or a simple Makefile to wrap commonly used west/cmake targets. Currently it's only possible to have folders at the top level.

Example of workspace layout:

west-workspace
├── README      # <- copied/linked from application/
├── application
│   ├── README

Suggested solution

Implement support for two new keywords in the projects section of a manifest:

• copyfile <src> <dst>
• linkfile <src> <dst>

Where <src> is relative to the path of the project being specified and <dst> is relative to the workspace.

copyfile creates a hard copy of a file linkfile creates a soft symbolic link

It shall be possible to specify multiple copyfile and linkfile directives for each project.

Example manifest:

manifest:
  projects:
    - name: zephyr
      copyfile: README.rst ./
      copyfile: LICENSE ./
      linkfile: west.yml ./
      linkfile: VERSION ./

[chrta]

This is also interesting for files like .editorconfig or .clang-format so they can be used from other sub-folders.

[mbolivar]

Hello, and thank you very much for taking the time to file an issue.

It shall be possible to specify multiple copyfile and linkfile directives for each project.

Example manifest:

manifest:
  projects:
    - name: zephyr
      copyfile: README.rst ./
      copyfile: LICENSE ./
      linkfile: west.yml ./
      linkfile: VERSION ./

I believe YAML lists, e.g. as supported by the import name-whitelist and other similar manifest syntax, may be easier to use.

Example:

manifest:
  # ...
  projects:
    - name: foo
      copyfile: README.txt .
    - name: bar
      copyfile:
        - Makefile .
        - README-makefile.txt .

I.e. copyfile and linkfile would accept either a string or a list of strings.

Thoughts?

[mbolivar]

I also want to discuss another couple of questions:

1. what should west do if the destination already exists?

My guess is west should not copy or link over an existing file, to avoid destroying people's work. This makes copyfile less useful, since e.g. README.txt would have to say "please read zephyr/README.rst for details" as west would not keep it up to date.

2. what should west do with linkfile on windows?

My guess is west should just emit a warning that it's not going to make the link, since Windows symlinks require admin access and are generally not well supported.

[tknapstad]

YAML lists

Using YAML lists for copyfile/linkfile has my vote, that is much more readable.

west update

I didn't think through the update issues yesterday, but I have a proposal for a solution that would support them:

1. When README.rst is first copied, compute a md5/sha256sum and store it in .west/checksums/README.rst.md5
2. When west update is executed, run the following steps:
   1. Calculate the checksum of the copied file, <dst> and compare to the checksum in .west/checksums. If the files differ, issue a warning that you have local modifications and should either remove <dst> to discard them or update <src> and commit it to the relevant Git repository. If the checksums are identical we can safely allow an update.
   2. After west has updated a repository, checksum all copyfile <src> files and compare them to the stored checksum in .west/checksums.
      • If they are different, copy the updated file if allowed by the previous step. If <dst> has local modifications, issue another warning that <src> has been updated, but <dst> contains local changes and can not be updated.
      • If there is no existing checksum in .west/checksums we can safely copy the file
      • Do a reverse check: If there are checksummed files in .west/checksums that are no longer listed in copyfile entries in the manifest, issue a warning that <dst> is obsolete.

To easily clean up any local modifications or obsolete copied files, maybe a -c/--clean option can be added to the west update command?

This more or less emulates Git behaviour and AFAICS also eliminates the need for the linkfile directive. If so, the Windows symlink issue also disappears.

[aescolar]

@mbolivar Just as extra motivation for the feature, that this is equivalent to repos' https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.md#Element-linkfile  and https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.md#element-copyfile

And would also benefit a project like BabbleSim (which today uses repo), and where the Makefile is made available at the top for users convenience: https://github.com/BabbleSim/manifest/blob/master/zephyr_docker.xml#L12

[c0fec0de]

We did a re-implementation of west: Git Workspace - an extension to git: https://pypi.org/project/git-ws It supports symlinks and copyfiles.