btile: alternative XOR storage of tile data for better compression ratio

[jorgegv]

Bedazzle user in SC forum raised the following issue:

I heard from other people (but didn't tried myself), that xoring data with previous byte is producing more similar bytes, and later compression gets better result.

Analysis:

I found something about this, it looks like for bytes that do not change very much (one from the next) it would work: https://stackoverflow.com/questions/133 ... -technique

If we do this we can't address the data directly, but only in a sequential manner, since I need to read each byte and then xor it with the previous one to get the good data.

I'm not really sure if this would be applicable to tile compression BUT if the 8-byte tile is stored top to bottom (which it normally is) it may work: normally there tends to be graphical continuity from top to down, which translates in byte similarities between one byte and the next and thus XORing may be useful as indicated in the link above.

It sounds very reasonable, and even usable for individual tiles: if the 8 bytes of a tile are always accessed one after another (they would unless using some non-common interleaving of graphic data), the overhead of a XOR to get each byte may be small and acceptable if the compression ratio gets better.

We should try storing the tiles with this schema, and then using ZX0 to compress the resulting DATASET, then compare results.

[TheStruggleForAntiSpaghetti]

I was about to point you to the SpectrumComputing forums and say: You could also discuss this idea with Einar, the dev of ZX7 and ZX0 as he's always very helpful and open for exchanging ideas.

But I see you're already exchanging ideas over there :)

On Mon, 10 Oct 2022 at 20:47, jorgegv @.***> wrote:

Bedazzle user in SC forum raised the following issue:

I heard from other people (but didn't tried myself), that xoring data with previous byte is producing more similar bytes, and later compression gets better result.

Analysis:

I found something about this, it looks like for bytes that do not change very much (one from the next) it would work: https://stackoverflow.com/questions/133 ... -technique https://stackoverflow.com/questions/13349278/looking-for-a-better-compression-technique

If we do this we can't address the data directly, but only in a sequential manner, since I need to read each byte and then xor it with the previous one to get the good data.

I'm not really sure if this would be applicable to tile compression BUT if the 8-byte tile is stored top to bottom (which it normally is) it may work: normally there tends to be graphical continuity from top to down, which translates in byte similarities between one byte and the next and thus XORing may be useful as indicated in the link above.

It sounds very reasonable, and even usable for individual tiles: if the 8 bytes of a tile are always accessed one after another (they would unless using some non-common interleaving of graphic data), the overhead of a XOR to get each byte may be small and acceptable if the compression ratio gets better.

We should try storing the tiles with this schema, and then using ZX0 to compress the resulting DATASET, then compare results.

— Reply to this email directly, view it on GitHub https://github.com/jorgegv/rage1/issues/109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAYZ6I62FQRSCOXKFSSNLDWCRQFZANCNFSM6AAAAAARBS5SF4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>