nydus-snapshotter icon indicating copy to clipboard operation
nydus-snapshotter copied to clipboard

rethink nydusd configuration file

Open imeoer opened this issue 1 year ago • 7 comments

  1. Nydusd configuration file includes registry auth information, it shouldn't be saved to disk (how to handle live upgrade and failover?).
  2. When the nydusd configuration file changes, the new nydusd should use the updated.
  3. Once the auth subscribed from k8s secret/docker config changes, nydusd also needs to update, in case of using an expired auth.

imeoer avatar Feb 28 '23 07:02 imeoer

When the nydusd configuration file changes, the new nydusd should use the updated.

nydus-snapshotter is providing the new nydusd config file to nydusd now

changweige avatar Mar 05 '23 02:03 changweige

  1. Nydusd configuration file includes registry auth information, it shouldn't be saved to disk (how to handle live upgrade and failover?).
  2. When the nydusd configuration file changes, the new nydusd should use the updated.
  3. Once the auth subscribed from k8s secret/docker config changes, nydusd also needs to update, in case of using an expired auth.

I am Guijie Wang, and I will complete this part of the work.

DarkMountain-wyz avatar May 31 '23 02:05 DarkMountain-wyz

When the nydusd configuration file changes, the new nydusd should use the updated.

The already running nydusd should not change its configuration I suppose. The newly created nydusd can use the new nydusd configuration. Otherwise, it's not easy to handle the consistency between nydusd config file and nydusd DB records

Can we only add an interface or credential service to nydus-snapshotter? The credential service can provide auth to nydusd when it requests

changweige avatar Jul 13 '23 06:07 changweige

The already running nydusd should not change its configuration I suppose. The newly created nydusd can use the new nydusd configuration. Otherwise, it's not easy to handle the consistency between nydusd config file and nydusd DB records

Yes, we said "the new nydusd should use the updated.".

Can we only add an interface or credential service to nydus-snapshotter? The credential service can provide auth to nydusd when it requests

The registry credential is the one of wanting to be updated for nydusd, the new nydusd maybe also need the new prefetch configuration, etc. I think we'd better not let nydusd become aware of the credential, it should be handled by the containerd credential plugin with snapshotter in the future, and then nydusd only get the full configuration from snapshotter.

imeoer avatar Jul 13 '23 12:07 imeoer

The already running nydusd should not change its configuration I suppose. The newly created nydusd can use the new nydusd configuration. Otherwise, it's not easy to handle the consistency between nydusd config file and nydusd DB records

Yes, we said "the new nydusd should use the updated.".

Can we only add an interface or credential service to nydus-snapshotter? The credential service can provide auth to nydusd when it requests

The registry credential is the one of wanting to be updated for nydusd, the new nydusd maybe also need the new prefetch configuration, etc. I think we'd better not let nydusd become aware of the credential, it should be handled by the containerd credential plugin with snapshotter in the future, and then nydusd only get the full configuration from snapshotter.

The registry auth ever passed to nydusd might be expired due to the registry configuration. So I suppose we need a way to refresh the auth nydusd is using.

changweige avatar Jul 14 '23 02:07 changweige

The already running nydusd should not change its configuration I suppose. The newly created nydusd can use the new nydusd configuration. Otherwise, it's not easy to handle the consistency between nydusd config file and nydusd DB records

Yes, we said "the new nydusd should use the updated.".

Can we only add an interface or credential service to nydus-snapshotter? The credential service can provide auth to nydusd when it requests

The registry credential is the one of wanting to be updated for nydusd, the new nydusd maybe also need the new prefetch configuration, etc. I think we'd better not let nydusd become aware of the credential, it should be handled by the containerd credential plugin with snapshotter in the future, and then nydusd only get the full configuration from snapshotter.

Moreover, I don't think the prefetch configurations have to update for an ever-started nydusd which had finished its prefetch jobs.

changweige avatar Jul 14 '23 03:07 changweige

  1. Once the auth subscribed from k8s secret/docker config changes, nydusd also needs to update, in case of using an expired auth.

Can we make nydus-snapshotter to send the updates to nydusd via a (new) nydusd API? Then there is no need for nydusd to connect to snapshotter UDS.

bergwolf avatar Nov 01 '23 04:11 bergwolf