treewide: silent download, unpack and others

[Cynerd]

This introduces changes from NuttX commit 03b164f59ce40a3f5677b0588af2aee8d9697bf6.

The only left out is 'Register:' as that provides pretty handly list of applications in the firmware.

The download is unified to handle it always the same way.

Follow up on https://github.com/apache/nuttx-apps/pull/1588#event-8649968171.

[Cynerd]

Please, wait with merge. There seems to be issues with downloading wamr. I have to look into it.

[acassis]

@Cynerd please let me know: does this PR hides the Downloading and Patching lines? I think it is important to users know that the files are downloaded during the building phase.

[ghost]

I agree. it is critical to know that a build cannot be completed while staying offline.

[hartmannathan]

I agree. it is critical to know that a build cannot be completed while staying offline.

@Cynerd please let me know: does this PR hides the Downloading and Patching lines? I think it is important to users know that the files are downloaded during the building phase.

I agree also. It is definitely critical to know that things are being downloaded.

Wherever we are doing a download, we should also allow a custom URL in Kconfig, so that developers who want to locally host the dependencies will be able to do so.

[hartmannathan]

Please, wait with merge. There seems to be issues with downloading wamr. I have to look into it.

You can mark the PR as Draft until ready for review. That will prevent someone merging by mistake...

[acassis]

Hi @Cynerd maybe a better approach is just show: Downloading: filename-x.y.z and when it finishes download: done.

What do you think?

[xiaoxiang781216]

@Cynerd could you fix the conflict?

[Cynerd]

I will look at this next week. This is pretty old and was created only to keep it consistent in my eyes with the rest of the changes introduced at that time. I think that it still makes sense in terms of echoing that the download is happening but not in the way of reporting its progress.

In the meantime, we also modified the build process of our own projects in a way that nothing is being downloaded, and thus, I had less incentive to deal with this.

[acassis]

I will look at this next week. This is pretty old and was created only to keep it consistent in my eyes with the rest of the changes introduced at that time. I think that it still makes sense in terms of echoing that the download is happening but not in the way of reporting its progress.

In the meantime, we also modified the build process of our own projects in a way that nothing is being downloaded, and thus, I had less incentive to deal with this.

Agree! All user friendly improvements are welcome!

[cederom]

ping :-)

[Cynerd]

I am kind of busy with other work, and I am failing to get some free time to finish this maintenance work. If anyone has free time on its hands, feel free to finish it. I just can't promise to look at it in a few weeks' time (I have been postponing it already for half a year, and the target now diverged so much that it must be started almost from scratch).

[cederom]

ping :-)

[linguini1]

@Cynerd any plans to continue with this PR? Otherwise I think we should close it.

[hartmannathan]

@Cynerd any plans to continue with this PR? Otherwise I think we should close it.

@linguini1 @cederom @acassis @xiaoxiang781216

IMHO we should never silently download external packages!

Why:

1. Security reasons: some developers need to ensure only known bits are used in the build.

2. Offline / airgapped build environments: some developers will need to manually obtain external packages.

3. Customization: some developers will want to customize the external packages, use alternative versions or branches, or supply their own substitute.

In fact I think in Kconfig we should have a global permission switch, similar to "allow GPL" / "allow BSD", called "Allow downloads" (CONFIG_ALLOW_DOWNLOADS). All invocations of curl, wget, git, or other remote download commands should be gated behind this switch to avoid the unwanted accidental download.

[linguini1]

Makes sense to me!

[xiaoxiang781216]

@Cynerd any plans to continue with this PR? Otherwise I think we should close it.

@linguini1 @cederom @acassis @xiaoxiang781216

IMHO we should never silently download external packages!

Why:

1. Security reasons: some developers need to ensure only known bits are used in the build.

2. Offline / airgapped build environments: some developers will need to manually obtain external packages.

3. Customization: some developers will want to customize the external packages, use alternative versions or branches, or supply their own substitute.

In fact I think in Kconfig we should have a global permission switch, similar to "allow GPL" / "allow BSD", called "Allow downloads" (CONFIG_ALLOW_DOWNLOADS). All invocations of curl, wget, git, or other remote download commands should be gated behind this switch to avoid the unwanted accidental download. we already have all these in Kconfig: https://github.com/apache/nuttx/blob/master/Kconfig#L28-L100 And all 3rd party library isn't enabled by default. If some package is downloaded from the network, it must be enabled with the license by user explicitly, so it's fine to skip the log without -v.

[cederom]

@hartmannathan: @linguini1 @cederom @acassis @xiaoxiang781216

IMHO we should never silently download external packages!

Why:

1. Security reasons: some developers need to ensure only known bits are used in the build.

2. Offline / airgapped build environments: some developers will need to manually obtain external packages.

3. Customization: some developers will want to customize the external packages, use alternative versions or branches, or supply their own substitute.

In fact I think in Kconfig we should have a global permission switch, similar to "allow GPL" / "allow BSD", called "Allow downloads" (CONFIG_ALLOW_DOWNLOADS). All invocations of curl, wget, git, or other remote download commands should be gated behind this switch to avoid the unwanted accidental download.

Full agree here thanks @hartmannathan ! Just like in FreeBSD Ports :-) Security + customization + testing -> to get selected packages before the build phase! :-)

++ as @xiaoxiang781216 noticed packages must match selected licensing model :-)

[Cynerd]

The point of this was not to hide the info about the downloaded package. That is just not what this does. It instead replaces the download progress with a message that is part of the existing NuttX reporting functionality that was introduced in the NuttX build system. You are still informed that the download is happening, but standard NuttX build progress reporting is not broken by the download progress.

Honestly, I would prefer that nothing be downloaded in the build phase, but that would require probably an additional phase to be added in NuttX itself (if that wasn't done in the meantime).

I haven't invested any time in this the past year, so I would just close this. As I already once stated, it is treewide, and because every package manages downloads in its own separate way, it is a lot of work to do this switch. I should have probably only added defines and done the change step by step. In the meantime, I saw that NuttX now has DOWNLOAD defined, so hopefully packages will migrate on their own, and thus this PR introduces absolutely nothing new.

[cederom]

Thank you @Cynerd ! Yup it would be best to have list of all stuff to be downloaded (aka "fetch phase") before the "buid phase".. we need to add something like this in a global design manner :-)