Developing a better compression system for the .fset files

[kalwalt]

I already posted this question in the NFT-Marker-Creator issue tracker. To keep tracks and find resources, solutions or anything else.

[kalwalt]

@nicolocarpignoli @Carnaux @ThorstenBux i think we could try to compress the files with https://github.com/kripken/lzma.js (it use LZMA compression algorithm) maybe packaged all together and then de-compressed inside jsartoolkit5? Do you think that it worth the try?

[ThorstenBux]

@kalwalt great idea. I was thinking about something like that but haven't started yet to look into possible options. Having that includes into jsartoolkit makes total sense to me.

[kalwalt]

So do you think that it could be a good idea? i know that artoolkit5 has zip.c inside ARUtil but i think that the LZMA has a better compression. I will start with this library instead. I will see what i can do!

[ThorstenBux]

Yes good idea for sure. Two things, 1 yes zip.c is included but I think what we can handle in JS is preferred to using c/wasm code. 2 Browsers might be handling decompress zipped artifacts automatically. That is one thing to consider before looking into doing it manually.

[kalwalt]

Yes good idea for sure.

ok! :smile:

Browsers might be handling decompress zipped artifacts automatically. That is one thing to consider before looking into doing it manually.

So doesn't it needed to decompress them? or are you saying that can be an issue?

[ThorstenBux]

Yes good idea for sure.

ok! 😄

Browsers might be handling decompress zipped artifacts automatically. That is one thing to consider before looking into doing it manually.

So doesn't it needed to decompress them? or are you saying that can be an issue?

Saying the browser can potentially decompress them automatically which is a great feature and could save you some work.

[kalwalt]

Yes good idea for sure.

ok! smile

Browsers might be handling decompress zipped artifacts automatically. That is one thing to consider before looking into doing it manually.

So doesn't it needed to decompress them? or are you saying that can be an issue?

Saying the browser can potentially decompress them automatically which is a great feature and could save you some work.

i understood now, thank you!

[ThorstenBux]

https://stackoverflow.com/questions/32172704/is-gzip-automatically-decompressed-by-browser

[ThorstenBux]

Hi there, how are you. I looked a bit into that today and found that compressing all artifacts (fset, fset3, iset) into one zip won't work as a browser can't extract one zip into multiple files. However, what we can do. We can take the content of the artifact and encode them with base64. Then add them to a json like this:

{
 fset: <<base64>>,
fset3: <<base64>>,
iset: <<base64>>
}

And would have in one file which potentially could be zipped. This would also have the extended benefit to easily add it to ARStudio. The NFT creator would need to be modified to allow this.

[ThorstenBux]

@carnaux what do you think?

[kalwalt]

I think that we can do it, or at least we can try.

[ThorstenBux]

We can do it :). The base64 would work I'm already implementing it.

[kalwalt]

Are you testing on the NFT-Marker-Creator? or directly on jsartoolkit5?

[ThorstenBux]

adding the implementation to jsartoolkit5. Once I have that working I change the Marker-Creator to create the JSON file.

[ThorstenBux]

I create the JSON manually for now.

[kalwalt]

Ok this sounds good!

[ThorstenBux]

Had a longer experimentation with my idea yesterday. Whereas it works and would allow the files to be contains in one file it does increase the file size by 33% due to the base64 encoding. I think that is not the intention what we are aiming for. For arStudio I would like to have the files within one file but increasing the size by 1/3 isn't suitable either. Will think about another solution.

[kalwalt]

Definetely has no sense, but nice idea, yes we have to think for another solution...

[Carnaux]

I think I can make it work using ZLIB(minizip) in C to compress files from memory and save a zip file and then use the same lib to decompress it, I was trying it earlier this week, but I had to do other things. It is in my to-do list, I will keep you guys updated!

[ThorstenBux]

Thank you @Carnaux. One thing to keep in mind though is that on the client (I'm thinking mobile device) the power to decompress might be limited and it needs to be possible to compile it with EM.

One other idea is. If we could merge the three files into one with delimiters that are parsed on C side then the server can served a zipped version and the browser would do the unzipping for us automatically. However, I don't know the overhead of parsing of such a combined binary file.

[Carnaux]

I will make some benchmarks of performance in PCs and Mobiles. The testing was already with emscripten, but was having some linking issues. When i was researching i found something about compressing the binary with RLE and then storing it as base32, maybe it works for the 3 in 1 idea.

[Carnaux]

Hey guys, good news, I got the compression to work, in C and with emscripten. Soon I will create a new repo for you to test it. But the preliminary results show a reduction of about 27%(testing with the pinball image) from 1246KB to 1042Kb. It works this way:

I wrote two codes(separate files), one to compress and one to decompress.

The compression code:

1 - I read the marker files and convert the raw buffer to a Hex string

  <Buffer 61 62 63 .....> to "616263"

2 - Create an object with the contents and stringfy it

let obj = { iset : "hex string", fset: "hex string", fset3: "hex string" } to {iset : "hex string", fset: "hex string",fset3: "hex string"}

I'm using hex strings instead of the buffer, because, on the stringfy, the UTF-8 encoding may compromise part of the data.

3 - Just send it to the C function that compresses and creates a .bin file that we will use as marker file( We can change the extension to whatever name we want, like .arMarker, let's choose a name for it)

Then the decompression code:

1- Reads the .binMarker file and writes a temp bin file to Module memory. I was having some issues with sending the contents as a string directly to the C code decompress it.

2- After reading the file and decompressing it we have the content string:

{iset : "hex string", fset: "hex string",fset3: "hex string"}

Now I'm writing the code to parse it and get each hex string and then load into artoolkit. I'm just worried about memory allocation, it works all fine, but it may cause problems in the future when merging with artoolkit or something like it.

[Carnaux]

By the way, I'm testing with Node for now, and it all happens within ms, for now, its really fast

[ThorstenBux]

Awesome, work. Love it. Me and @kalwalt we’re thinking of moving jsartoolkit related stuff to a dedicated organisation on GitHub I think your tools would be awesome to live there too. What do you think?

Regarding the name suffix. How about .nft? Or .zft for compressed nft😊.

I’m also setting up a public web service to run the nft creation on which is very fast. Created nfts will be stored on a cloud storage and can be downloaded from there to keep it anyc and integrate it with webarstudio. Basically reusing all of your code and rearrange it a bit to allow for aws lambda deployment. 😊

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Daniel Fernandes Gonçalves de Oliveira [email protected] Sent: Thursday, April 23, 2020 12:45:33 PM To: kalwalt/jsartoolkit5 [email protected] Cc: Thorsten Bux [email protected]; Mention [email protected] Subject: Re: [kalwalt/jsartoolkit5] Developing a better compression system for the .fset files (#34)

By the way, I'm testing with Node for now, and it all happens within ms, for now, its really fast

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkalwalt%2Fjsartoolkit5%2Fissues%2F34%23issuecomment-618112210&data=02%7C01%7C%7Cee209f2002644658a11908d7e71fa212%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637231995349652552&sdata=KxMSvhhMUKmMbJztn7YRH3PRRJQxpDMpRAEYl3psLpE%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PHNSQJH222KBBXLOI3RN6FS3ANCNFSM4KLRXNRA&data=02%7C01%7C%7Cee209f2002644658a11908d7e71fa212%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637231995349662546&sdata=j8Cdwe2Eda19rCcxmGb0QnaV8ROzni%2FnDVtMbkN4Kvo%3D&reserved=0.

[Carnaux]

It would be nice! If you need help with the new organization or with the web service, just tell me!

I love .zft, I'm gonna change to it now.

[kalwalt]

That's great @ThorstenBux @Carnaux !! I vote for .zft can't wait to test this!

[Carnaux]

https://github.com/Carnaux/ZFT-Creator the testing version is uploaded! Now I'm going to work with the browser version. I will read the Artoolkit docs to learn how it is loading the markers and then integrate them.

[Carnaux]

After you read the C code, let's discuss the memory management.

[kalwalt]

@Carnaux I will test it!

[Carnaux]

@kalwalt @ThorstenBux It is done added the browser version, we now just have to implement it in artoolkit!