Does libpmemobj or libpmemobj-cpp support dev-dax mode?

[GaoDong666]

QUESTION:

Does libpmemobj or libpmemobj-cpp support dev-dax mode?

Details

Hello, I would like to program in devdax mode, I wonder if libpmemobj or libpmemobj-cpp supports devdax mode or not. Are there any programming examples of devdax mode? Thanks!

[lukaszstolarczuk]

Hi, Thanks for posting your question!

I'm convinced libpmemobj (from PMDK repository) is regularly tested on dax devices. Since libpmemobj-cpp is a C++ wrapper for that library, we expect it to work as well... I'm just afraid we haven't fully tested that for a while... but I just executed a simple example on my machine with /dev/dax1.0 device and it worked like a charm.

Because we use pmempool to handle pmem pools in libpmemobj(-cpp) it is as simple as changing the path to your file/pool (from fsdax to devdax file). I've tested inline_string example from our repository and just passed as its parameter a simple path /dev/dax1.0/somefile. No more changes were required.

If you need any more assistance, don't hesitate to ask.

[GaoDong666]

@lukaszstolarczuk Thanks for your reply! I'm new to PM and working on a small database running in fsdax mode; it works with a number of interfaces of libpmemobj(-cpp), such as:

#include <libpmemobj++/container/array.hpp> #include <libpmemobj++/container/vector.hpp> #include <libpmemobj++/utils.hpp> #include <libpmemobj++/shared_mutex.hpp>

Since I need huge pages (1 GB) mapping in some cases, I wonder if I could run the database in devdax mode without a lot of changes. In other words, I would like to use the advanced programming interfaces provided by libpmemobj-cpp in devdax mode. Do you know of any projects working in devdax mode? I would like to learn how they work.

[GaoDong666]

As far as I know, the biggest difference between fsdax and devdax is whether the file system is needed or not. The devdax model may be more flexible; but due to its lack of file system support, programmers need to consider more details when programming. Therefore, I wonder if devdax can be used in complex software systems such as databases.

[GaoDong666]

I tested inline_string example on my machine with simulated PM (8GB fsdax & 8GB devdax) and passed in fsdax mode but failed in devdax mode. The command I entered and the corresponding results are as follows:

root@MS-7D31:/home/gaodong/WorkSpace/test_dax# pmempool create obj --layout=inline_string -s 100M /mnt/pmem0/test_pool_fsdax
root@MS-7D31:/home/gaodong/WorkSpace/test_dax# ./inline_string /mnt/pmem0/test_pool_fsdax 
example
some new, longer value
root@MS-7D31:/home/gaodong/WorkSpace/test_dax# ./inline_string /mnt/pmem0/test_pool_fsdax 
some new, longer value
some new, longer value
root@MS-7D31:/home/gaodong/WorkSpace/test_dax# 
root@MS-7D31:/home/gaodong/WorkSpace/test_dax# 
root@MS-7D31:/home/gaodong/WorkSpace/test_dax# pmempool create obj --layout=inline_string -s 100M /dev/dax1.0/test_pool_devdax
root@MS-7D31:/home/gaodong/WorkSpace/test_dax# ./inline_string /dev/dax1.0/test_pool_devdax
Failed opening pool: os_access "/dev/dax1.0/test_pool_devdax": Not a directory
To create pool run: pmempool create obj --layout=inline_string -s 100M path_to_pool
Pool already closed
root@MS-7D31:/home/gaodong/WorkSpace/test_dax#

No errors occurred while creating the persistent memory pools. However, an error occurred when attempting to read or write to the devdax pool. I tried not specifying the size of the devdax pool while creating, but that didn't help.

My simulated namespaces are as follows:

root@MS-7D31:/home/gaodong/WorkSpace/test_dax# ndctl list
[
  {
    "dev":"namespace1.0",
    "mode":"devdax",
    "map":"dev",
    "size":8453619712,
    "uuid":"0cf67d94-62e9-4e90-a01c-87ab6a187ef1",
    "chardev":"dax1.0",
    "align":2097152
  },
  {
    "dev":"namespace0.0",
    "mode":"fsdax",
    "map":"dev",
    "size":8453619712,
    "uuid":"4836bef5-aed4-4377-944c-e014f7560165",
    "sector_size":512,
    "align":2097152,
    "blockdev":"pmem0"
  }
]
root@MS-7D31:/home/gaodong/WorkSpace/test_dax#

Did I make any mistake or miss any steps?

[lukaszstolarczuk]

As for the example run, did you double check the path of dev-dax pool - is the /dev/dax1.0/test_pool_devdax correct path? and is it a file? does it exist after pmempool command execution?

Since you're new to PM, I recommend you to try and read our book about Persistent Memory - you can find it on our pmem.io website; beside the book, there are other docs, guides, and blog posts you may find useful.

I believe both, fsdax and devdax will support 1GB mappings; right now, while using libpmemobj(-cpp) the limit for a single object's mapping is nearly 16GB. Generally speaking, both should do fine in complex software systems.

I'm afraid I'm not aware of any projects using dev-dax. On the libpmemobj(-cpp) level, the pool file is handled rather seamlessly and it should not affect your application. I believe there might be some use cases, where you could take advantage of dev-dax pools (not having filesystems) - perhaps, please see Chapter 3 in the mentioned book, for more details on this topic.