Cannot create Hyper Files on mounted webdav volumes despite full write permission.

[DrMaphuse]

Problem Description

When trying to create a hyper file using tableauhyperapi, on a webdav volume mounted via davfs2 (this might apply to other file systems as well), there seems to be some issue writing the file.

The permissions per se are not a problem, as you can see in the sample code:

Sample Code

from tablefrom pathlib import Path

from tableauhyperapi import Connection, CreateMode, HyperProcess, Telemetry

test_file = Path("/mnt/data/test.hyper")

# write and read file to test write permissions
test_file.unlink(missing_ok=True)  # delete existing file
with test_file.open("w") as f:
    f.write("Write/read success!")
with test_file.open("r") as f:
    print(f.readline())
test_file.unlink(missing_ok=True)  # delete test file before testing hyper

with HyperProcess(
    telemetry=Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU,
) as hyper:
    # Open a connection to the Hyper process. This will also create the new Hyper file.
    # `CREATE_AND_REPLACE` mode causes the file to be replaced if it already exists.
    with Connection(
        endpoint=hyper.endpoint,
        database=test_file,
        create_mode=CreateMode.CREATE_AND_REPLACE,
    ) as connection:
        print("Hyper write success!")

---------------------------------------------------------------------------
HyperException                            Traceback (most recent call last)
Input In [2], in <cell line: 15>()
     13 test_file.unlink(missing_ok=True)  # delete test file before testing hyper
     15 with HyperProcess(
     16     telemetry=Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU,
     17 ) as hyper:
     18     # Open a connection to the Hyper process. This will also create the new Hyper file.
     19     # `CREATE_AND_REPLACE` mode causes the file to be replaced if it already exists.
---> 20     with Connection(
     21         endpoint=hyper.endpoint,
     22         database=test_file,
     23         create_mode=CreateMode.CREATE_AND_REPLACE,
     24     ) as connection:
     25         print("Hyper write success!")
     26         pass

File ~/.local/lib/python3.9/site-packages/tableauhyperapi/connection.py:99, in Connection.__init__(self, endpoint, database, create_mode, parameters)
     96     database = parameters['dbname']
     97     del parameters['dbname']
---> 99 self.__cdata = self.__create_connection(endpoint, database, create_mode, parameters)
    100 self.__endpoint = endpoint
    102 # Lock to serialize cancel() and close() calls.

File ~/.local/lib/python3.9/site-packages/tableauhyperapi/connection.py:126, in Connection.__create_connection(endpoint, database, create_mode, parameters)
    123         native_params.set_value(key, value)
    125 pp = ffi.new('hyper_connection_t**')
--> 126 Error.check(hapi.hyper_connect(native_params.cdata, pp, create_mode.value))
    127 return ffi.gc(pp[0], hapi.hyper_disconnect)

File ~/.local/lib/python3.9/site-packages/tableauhyperapi/impl/dllutil.py:100, in Error.check(p)
     97 if p != ffi.NULL:
     98     # this will free the error when it goes out of scope
     99     errp = Error(p)
--> 100     raise errp.to_exception()

HyperException: The database "hyper.file:/mnt/data/test.hyper" could not be created: Growing the database file failed
Context: 0xfa6b0e2f

Versions:

System: Ubuntu 22.04.1 LTS Python: 3.9.7 tableauhyperapi: 0.0.15530

[jonas-eckhardt]

@DrMaphuse can you please include the log file?

Thanks Jonas

[DrMaphuse]

Sure! Can you narrow it down to specific sections of the log? As it is, the log contains some sensitive information, which means that I can't put it online.

Edit: After a quick glance at the log I found the following section - does this help?:

"query-end-user-error","v":{"error-code":"42704","error-message":"The database \"hyper.file:/.../timeline.hyper\" could not be created: Growing the database file failed","error-detail-internal":"file: \"/.../timeline.hyper\"\nsystem error: No such device"

[jkammerer]

Unfortunately, I don't think that I will be able to reproduce this quickly with my setup. If you would be willing to share the log via private channels we could get in touch on Slack or on the Tableau Community forum: https://community.tableau.com/s/question/0D54T00000C5OqVSAV/join-the-datadev-slack

[srujanV123]

were you able to figure out the issue ? we are getting the similar error : The database "hyper.file:/test/test.hyper" could not be created: The database file could not be created\nContext: 0xfa6b0e2f\n

[jkammerer]

We did not figure out the problem and are currenlty not looking into this since we need more information. If you could provide a hyperd Log file, it might help us getting to the bottom of this.

[DrMaphuse]

I can't reproduce the problem easily myself now, because we switched to another filesystem for other reasons, and that made the problem go away.

I suspect, however, that it had something to do with the webdav protocol per se. Webdav does not permit partial read/write operations - I imagine that this would make it difficult to "grow" a file piecewise, as Tableau appears to be doing.

Maybe it's possible to test the partial read/write access before writing the actual file and generate a corresponding error message?

[veruzi]

Got the same error today while trying to UNION multiple extracts. Used the code from here (the optimal one): https://engineering.tableau.com/using-the-hyper-api-to-union-hyper-files-3fd0904fd69b

I suspect the temp folder does not have enough available space. Any parameter to change the temp folder for hyperd ?

Error message below.

{"ts":"2023-07-12T02:22:15.340387","pid":49160,"tid":"292e8","sev":"error","req":"6","sess":"3vDu2Jx9TJS8pVPhDc7H5w","k":"objstore-read-from-main-resource","v":{"elapsed-setup-ioframe":3.94e-07,"id":{"category":"RelationDataBlock","part1":0,"part2":58,"part3":0,"part4":2609},"compression-method":"lz4","offset":24313932992,"storage-size":27877,"storage-id":{"provider":"file","path":"temp\SubscriptionsAnalytics-Alex\unpack\chunk_1.hyper"},"process-object":true,"retrieve-object":true,"elapsed-storage-resource-read":8.685e-06,"elapsed-validate":1.185e-06,"elapsed-decrypt":7.89e-07,"encryption-method":"none","size-uncompressed":133376,"elapsed-uncompress-get-size":5.527e-06,"elapsed-acquire-lock-for-alloc":7.9e-07,"exception":{"code":"53100","message":"Growing the database file failed","detail-internal":"file: "\\\\?\\C:\\Users\\veruzi\\AppData\\Local\\Temp\\10\\hyper_transient.49160.2.tmp"\nsystem error: SetFileInformationByHandle(): There is not enough space on the disk."},"error-during-access":false,"elapsed-read-ioframe":0.0134309,"elapsed":0.0217515}}

Thanks, Alex

[veruzi]

I think this parameter will change the temp directory: "hyper-temp-directory-override"