cluster/ceph: Fix built-in pool creation for Reef.

[markhpc]

Add the required flag to make default "." pools now required in Reef.

[markhpc]

This works fine in testing, but is going to require version_compat to be set for anything pre-reef. The UI change that was introduced is not backward compatible.

@neha-ojha @NitzanMordhai This will likely mean that any teuthology tests involving the creation of "." pools will fail on old verisons of Ceph unless version_compat is set to the right ceph release.

[neha-ojha]

This works fine in testing, but is going to require version_compat to be set for anything pre-reef. The UI change that was introduced is not backward compatible.

@neha-ojha @NitzanMordhai This will likely mean that any teuthology tests involving the creation of "." pools will fail on old verisons of Ceph unless version_compat is set to the right ceph release.

@NitzanMordhai can you please test this PR against rados/perf and make any necessary changes

[NitzanMordhai]

@NitzanMordhai can you please test this PR against rados/perf and make any necessary changes

@neha-ojha I had some issues with teuthology yesterday, i started to test it today, well.. the one test that i ran for quincy is green, but it didn't actually do anything. it will skip the directory (we are using use_existing: true in rados:perf) Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f http://pulpito.front.sepia.ceph.com/nmordech-2023-03-23_05:51:28-rados:perf-main-distro-default-smithi/7217342/

I think we should merge PR https://github.com/ceph/cbt/pull/277 first, @markhpc what do you think?

[lee-j-sanders]

I rebased this PR into the latest and greatest, and it fixes the issue I'm seeing with squid where you cannot do any RGW/Object based work with CBT because the .rgw.root pool creation fails. So this is a good fix, thanks!