PFS无法向Ceph的共享存储设备复制文件

[asfakkk]

我按照文档步骤安装PolarDB，PFS可以向我本地的磁盘/sdb创建文件夹并复制文件，初始化数据库。但我将设备转成我映射到ceph的设备时，我可以正常的使用PFS创建文件夹，但是我无法向该设备复制文件。 我可以正常使用PFS创建和浏览目录

[postgres@master01 PolarDB-for-PostgreSQL]$ sudo pfs -C disk ls /rbd0/
  File  1     4194304           Mon Jul 18 14:42:54 2022  .pfs-paxos
  File  1     1073741824        Mon Jul 18 14:43:10 2022  .pfs-journal
   Dir  1     0                 Mon Jul 18 14:44:35 2022  shared_data
total 2105344 (unit: 512Bytes)

ceph映射的共享设备

[root@master01 lsblk
NAME                                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                           8:0    0   500G  0 disk
├─sda1                                        8:1    0   200M  0 part /boot
└─sda2                                        8:2    0 499.8G  0 part
  ├─centos-root                             253:0    0 497.8G  0 lvm  /
  └─centos-swap                             253:1    0     2G  0 lvm  [SWAP]
sdb                                           8:16   0   500G  0 disk
└─ceph--9847355e--04ec--436e--b20e--afc810986aa9-osd--block--ca9419fd--3b15--4355--ac09--6b59b08e4511
                                            253:2    0   500G  0 lvm
sr0                                          11:0    1  1024M  0 rom
rbd0                                        252:0    0   125G  0 disk

无法复制文件

[postgres@master01 PolarDB-for-PostgreSQL]$ sudo pfs -C disk cp /home/postgres/PolarDB-for-PostgreSQL/config.log  /rbd0/shared_data/
pfs tool cmd record:cp /home/postgres/PolarDB-for-PostgreSQL/config.log /rbd0/shared_data/
[PFS_LOG] Jul 18 15:23:38.737896 INF [17101] pfs build version:libpfs_version_("pfsd-build-desc-_-Wed Jun 15 09:39:00 UTC 2022")
[PFS_LOG] Jul 18 15:23:38.737985 INF [17101] pid: 17096, caller: /bin/bash /var/tmp/sclxquNkA
[PFS_LOG] Jul 18 15:23:38.738033 INF [17101] pid: 17095, caller: scl enable devtoolset-9  'pfs' '-C' 'disk' 'cp' '/home/postgres/PolarDB-for-PostgreSQL/config.log' '/rbd0/shared_data/'
[PFS_LOG] Jul 18 15:23:38.738061 INF [17101] pid: 17094, caller: /usr/bin/sudo -E env LD_LIBRARY_PATH=/opt/rh/llvm-toolset-7.0/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib PATH=/home/postgres/tmp_basedir_polardb_pg_1100_bld/bin:/opt/rh/llvm-toolset-7.0/root/usr/bin:/opt/rh/llvm-toolset-7.0/root/usr/sbin:/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin scl enable devtoolset-9  'pfs' '-C' 'disk' 'cp' '/home/postgres/PolarDB-for-PostgreSQL/config.log' '/rbd0/shared_data/'
[PFS_LOG] Jul 18 15:23:38.738092 ERR [17101] cp: can't find any pbdname
[PFS_LOG] Jul 18 15:23:38.738098 ERR [17101] cp failed: impl specific
cp failed: impl specific

无法初始化服务

[postgres@master01 PolarDB-for-PostgreSQL]$ sudo $HOME/tmp_basedir_polardb_pg_1100_bld/bin/polar-initdb.sh  $HOME/primary/ /rbd0/shared_data/
pfs tool cmd record:cp -r /home/postgres/primary//base /rbd0/shared_data//
[PFS_LOG] Jul 18 15:19:55.741567 INF [17093] pfs build version:libpfs_version_("pfsd-build-desc-_-Wed Jun 15 09:39:00 UTC 2022")
[PFS_LOG] Jul 18 15:19:55.741655 INF [17093] pid: 17045, caller: /bin/bash /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/polar-initdb.sh /home/postgres/primary/ /rbd0/shared_data/
[PFS_LOG] Jul 18 15:19:55.741689 INF [17093] pid: 17040, caller: /bin/bash /var/tmp/sclQIyOn2
[PFS_LOG] Jul 18 15:19:55.741712 INF [17093] pid: 17039, caller: scl enable devtoolset-9  '/home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/polar-initdb.sh' '/home/postgres/primary/' '/rbd0/shared_data/'
[PFS_LOG] Jul 18 15:19:55.741737 INF [17093] pid: 17038, caller: /usr/bin/sudo -E env LD_LIBRARY_PATH=/opt/rh/llvm-toolset-7.0/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib PATH=/home/postgres/tmp_basedir_polardb_pg_1100_bld/bin:/opt/rh/llvm-toolset-7.0/root/usr/bin:/opt/rh/llvm-toolset-7.0/root/usr/sbin:/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin scl enable devtoolset-9  '/home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/polar-initdb.sh' '/home/postgres/primary/' '/rbd0/shared_data/'
[PFS_LOG] Jul 18 15:19:55.741769 ERR [17093] cp: can't find any pbdname
[PFS_LOG] Jul 18 15:19:55.741774 ERR [17093] cp failed: impl specific
cp failed: impl specific
copy /home/postgres/primary//base to /rbd0/shared_data// fail

...

[polardb-bot[bot]]

Hi @asfakkk ~ Thanks for opening this issue! 🎉

Please make sure you have provided enough information for subsequent discussion.

We will get back to you as soon as possible. ❤️

[mrdrivingduck]

@asfakkk

看起来您可能需要对您的块设备建立一个软链接，因为 PFS 只能支持对指定前缀的设备名进行操作。可以看下 这个文档。

---

Seems that you should create a symbolic link to your block device, because PFS only considers specific prefix as a valid device name. See docs here.

[mrdrivingduck]

@asfakkk Has the problem been solved?

[mrdrivingduck]

/close