Integrating a data upload feature as command

[f2cx]

Background

In the August cloud release, the new tool Qlik DataTransfer was highlighted. I have no idea which API is used in the background technically to do the data upload to the cloud-based Qlik Sense instance.

Description

I would love to have such "data upload" functionality in the great qlik-cli tool. Something like that:

qlik upload <localFile> --spaceId <spaceId>

Discussion

Are there any plans to make this possible with qlik-cli? While the Qlik Data Transfer tool is nice for Windows machines, it is not an option to use this in Linux server environments with lots of automation. There the usage of qlik-cli would be great!

Any comments from your side? @glooms @wennmo

[wennmo]

Hi @f2cx, thanks for reaching out!

This is an area that will be improved in qlik-cli going forward, and is something that is actively being worked on. The qlik-cli only exposes the public APIs of the SaaS solution, so unfortunately the same functionality as the Qlik Data Transfer tool is not available in the latest version.

[dnabb]

Hello,

Two years have passed since the original question, and now there is a data-file API which in my understanding should do what was requested in this issue.

Unfortunately though, I cannot find any example, and the documentation for this API (both the qlik-cli version and the REST version is quite concise when it comes to the --file parameter:

IFormFile form multipart/form-data

Is there any more documentation / examples on how to practically use the API?

[wennmo]

Hi @dnabb, agree that the documentation is lacking examples for the Data Files API and I will look into adding that. Thanks for highlighting it.

For uploading a file using the qlik-cli and the Data Files API: qlik data-file create --name=<name for the uploaded file> --file=<path to the file that should be uploaded>

Listing data-files: qlik data-file ls

Fetching a specific data file: qlik data-file get <data file ID>

Updating name of a data file: qlik data-file --name <updated name> update <data file ID>

Delete a data-file: qlik data-file rm <data file ID>

[dnabb]

Thanks @wennmo, this helped.

Some more questions that I got while playing with the Data Files API and that I would have liked find answers for in the documentation:

• qlik data-file update also takes a --file parameter (if I want to overwrite an existing file with another one I am uploading), but when using it to upload a file, it stops with an error after 500 MB. The same file can be uploaded without problem (and apparently faster) with qlik data-file create and then the original file can be overwritten with qlik data-file update --name <name of t the file to overwrite> --sourceId <id of the file we just uploaded>
• With the process I just described, I am basically creating a temp content, which I assume is what the --tempContentFileId is for, but what exactly is a temp content and how do I upload one? I see there is a REST endpoint, but it doesn't seem to be exposed in qlik-cli?
• qlik data-file quotas returns 3 different size limits (one of which is about 500 MB, see issue above with uploading large files) but what are the quotas, what do they impact and is there a way to, for example, mark an upload as "large file" (to avoid the 500MB limit when updating)?

[wennmo]

Thank you for the feedback @dnabb 🙇

You are correct that there is a quota for the size of the file in the data-file API, and files with a size that exceeds 500MB should actually be uploaded through the temporary-contents API. All of these pieces are not yet surfaced in qlik-cli unfortunately, but I will look into if we can add support for these scenarios in coming releases.

The temporary-contents API is actually being used in the qlik-cli for uploading of big files, which you should see if you provide the --verbose flag when uploading a big file. I will investigate why this is not working for the qlik data-file update command.

There is a tutorial for uploading and managing of data files in Qlik Sense SaaS which provides a bit more information, but this is definitely an area where the documentation needs improvement/clarification.

[withdave]

Closing as was completed in 2023.