zpool add should warn about adding a storage or special vdev to raidz

[tkittich]

Describe the feature would like to see added to OpenZFS

zpool add should warn about adding a storage or special vdev to a raidz pool that the added vdev cannot be removed later. zpool add should probably require -f or --allow-vdev-cant-be-removed flag in this case as well.

Also, the man page for zpool-remove is probably outdated. It states "Top-level vdevs can only be removed if the primary pool storage does not contain a top-level raidz vdev", which is not true because log and cache vdevs can be removed from a raidz pool.

How will this feature improve OpenZFS?

A warning about a storage or special vdev cannot be removed from a raidz pool would be very helpful and could prevent users from having to recreate the entire pool.

root@h00:/tmp/root# zpool add testpool special mirror /tmp/root/*.special
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool and new vdev with different redundancy, raidz and mirror vdevs, 2 vs. 1 (2-way)
>>**should be added**<< the storage or special vdev with raidz pool cannot be removed later

Additional context

There're many cases where users did not expect the vdevs couldn't be removed from raidz: #14312 , remove-or-migrate-special-vdev , why-doesnt-zfs-vdev-removal-work , remove_raidz_vdevs