SES5: osd.redeploy fails for filestore -> bluestore

[swiftgist]

Description of Issue/Question

For anybody reading this, the workaround is to remove and deploy the osd as two steps.

The comparison in osd.is_incorrect is relying on the pillar. I don't see a quick fix for this.

Here's one example, although many existed in this particular installation. The grains are correct for

       25:
            ----------
            fsid:
                0f85b6ed-cedb-4296-b35e-3cead398f688
            partitions:
                ----------
                journal:
                    /dev/disk/by-id/wwn-0x50014ee2600a9932-part2
                osd:
                    /dev/disk/by-id/wwn-0x50014ee2600a9932-part1

and lists a filestore OSD. Here are the corresponding snipped logs from an osd.redeploy run

[INFO    ] Partition: /dev/disk/by-id/wwn-0x50014ee2600a9932-part1
[INFO    ] readlink -f /dev/disk/by-id/wwn-0x50014ee2600a9932-part1
[DEBUG   ] '/dev/sdad1'
[DEBUG   ] ''
[DEBUG   ] splitting partition /dev/sdad1
[INFO    ] ID: 25
[INFO    ] Disk: /dev/sdad
...
[INFO    ] No journal specified for /dev/sdad
[DEBUG   ] OSD config: 
{'capacity': '3726 GB',
 'db': None,
 'db_size': None,
 'device': '/dev/sdad',
 'disk_format': 'bluestore',
 'encryption': False,
 'journal': False,
 'journal_size': '5242880K',
 'size': 4000787030016,
 'small': False,
 'tli': {'/dev/sda': {'format': 'bluestore'},
         '/dev/sdaa': {'format': 'bluestore'},
         '/dev/sdab': {'format': 'bluestore'},
         '/dev/sdac': {'format': 'bluestore'},
         '/dev/sdad': {'format': 'bluestore'},
         '/dev/sdae': {'format': 'bluestore'},
...

The is_incorrect uses the OSDConfig which inspects the pillar and not the actual disk. The pillar part is

               /dev/disk/by-id/wwn-0x50014ee2600a9932:
                    ----------
                    format:
                        bluestore

The end result is that all disks come back as "bluestore" which matches the desired configuration, so no OSD gets removed and recreated. The disk_format and journal above are correct for the pillar, but not the actual OSD.

This is related to bnc:1157549.