NFS: door does not react to pool address updates.

[paulmillar]

The NFS door creates a PoolDS object for each pool. This holds information about the pool, including the list of IP addresses. The PoolDS object is created when the pool is first used and is kept forever, or until the admin uses the pool reset id command.

If the pool network addresses change (e.g., an interface is brought up, or IP-address alias is added) then the door is notified of the change (via the PoolPassiveIoFileMessage message) but the set of addresses stored in any existing PoolDS object is not updated to match the updated information.

Using the pool reset id provides one work-around, as it removes the corresponding PoolDS object, but this has other side-effects and the command would need to be run for each pool.

Restarting the domain hosting the NFS door provides another work-around. This has, perhaps, even stronger side-effects.

[kofemann]

Indeed, the pool to IP multipath mapping is cached by the door and reused. Nevertheless, the pool can invalidate this mapping by updating it's verifier:

    PoolDS getOrCreateDS(String name, long verifier, InetSocketAddress[] poolAddress) {
        _wlock.lock();
        try {

            PoolDS ds = _poolNameToIpMap.get(name);
            if (ds != null && ds.getVerifier() == verifier) {
                return ds;
            }

            if (ds != null) {
                // remove old mapping
                _deviceMap.remove(ds.getDeviceId());
            }
            deviceid4 deviceid = deviceidOf(_nextDeviceID++);
            ds = new PoolDS(name, deviceid, poolAddress, verifier);
            _poolNameToIpMap.put(name, ds);
            _deviceMap.put(ds.getDeviceId(), ds);
            return ds;
        } finally {
            _wlock.unlock();
        }
    }

However, there are two issues:

1. The verifier is bound to pool start time and doesn't indicates network configuration change
2. NFS transfer service builds multipath list only once (well, typically we don't expect a pool to add a new interface dynamically.

If we want to support dynamic interfaces on pools, then we should start to support invalidating existing pNFS devices (data servers). For this dCache should issue callbacks to all clients as soon as a pool has changed the network configuration. Invalidating a device will invalidate all layouts associated with it. IOW, the layout lifecycle state machine becomes quite complicated.

Thus, we should go that path only if we expect dynamic interface addition to being a common use-case.

[paulmillar]

The specific scenario that triggered this issue is a one-off and not something that (I anticipate) would happen very often.

That said, the changes would involve updating a running system. Therefore, the dynamic address support (in the pool) is quite helpful, in this case. This means that networking changes don't require restarting all the pools.

Would simply binding the verifier to both the pool start time and the available pool addresses work?

Otherwise, I guess the anticipated cost/benefit doesn't justify the work.