Support importing/exporting Voxlap format (.vxl and .kv6)

[damienflament]

It should be great to allow importing/exporting files used by the games Ace of Spades/OpenSpades (which use the Voxlap engine).

The map files (*.vxl) might be too large for editing in Goxel (they are actually describing a volume of 512x512x64 voxels).

But the models files (*.kv6) are already editable with VoxelShop. Supporting this format in Goxel should be great !

[guillaumechereau]

This is the second time someone request the kv6 files support, so I guess I will have a look at it. Do you have any resources on the file format?

[damienflament]

As Ace of Spades is not open source, I looked in OpenSpades source code. The LoadKV6 function of the VoxelModel class get the KV6Header and the KV6Blocks.

You can also look the KV6Importer and KV6Exporter of VoxelShop.

If you want more information about the KV6 file format (maybe some documentation), you can ask to the OpenSpades maintainer. He answers quickly.

[guillaumechereau]

Thanks, I will have a look at that.

[guillaumechereau]

I just added basic support for kv6 files in the master branch.

[guillaumechereau]

And now also kvx.

[guillaumechereau]

Only loading for the moment.

[guillaumechereau]

And finally support for reading vxl files too:

[damienflament]

Nice !

If reading/modifying Ace of Spades/OpenSpades maps is easy and efficient, so you just made the new map editor !

Actually, mapping for those games is done using Voxed (proprietary, old, hard to use and only available on Windows) or in-game (not really easy as you can only move using the game physics).

[damienflament]

VXL files seems mirrored during importation:

You should read "for * Sauerkraut". Moreover, the distance of view seems limited (look at the truncated parts on the top of the screenshot (seen as white, but are transparent when exported as a PNG file).

[guillaumechereau]

I will try to fix those issues. Also I think the navigation with large models has to be improved, zooming is particularly bad for the moment.

[damienflament]

You are right about navigation in large models. But I think it will be another issue ticket.

I'm surprised that modifying large models does not require more resources than currently ! VoxelShop (which I used to modify OpenSpades models until now) does not accept large models.

Nice work!

[guillaumechereau]

Thanks, one of the goal for goxel is to support unbounded voxel models, so I made sure to optimize for this case, even though the interface is not very good in that regard.

[damienflament]

I made a comparison between Terravox and Goxel.

Terravox was made by the author of OpenSpades to make terrain height-maps. So, it handles VXL format natively (while writing this sentence, I think it's also a good reference about VXL format handling).

Supposing that the axes have the same orientation in both softwares and that, obviously, Terravox correctly handles the VXL file format, we can see that, in Goxel:

• the model is horizontally reversed (along the X axis)
• the model is vertically reversed (along the Z axis)

Concerning the Z axis, Terravox handles it as the Ace of Spades/OpenSpades games do it: The maximum in-game height is at level 0. Nothing can be constructed upper. The minimum height is at level 63 (water). I don't know if handling the Z axis the same way in Goxel for VXL files is relevant, as it's really specific to those games and that Goxel is a generic voxel editor.

[guillaumechereau]

I fixed the orientation and clipping bug. For the axis, for the moment prefer to stay with goxel default (Z up), though it might make sense to support different referential from a setting in the future. Terravox is quite good. It makes me want to add a 'grow' tool similar to what they have, that could be convenient to edit terrain maps.

[guillaumechereau]

I just added support for saving as vxl. It's totally untested, so there might be bugs or corrupted files.

[guillaumechereau]

I'll close this issue not that we have basic support. If there are specific bugs with voxlap export it will be better to open a new issue for each problem.

[OverwhelmingFire]

Why are .vxl maps not saved properly? I'm new to Goxel, started using it mainly to create maps for AoS. The problem is the following: when I import a working map in the Goxel, adjust the dimensions and export it back into a new .vxl, I get a file with size completely different from the original one (see the screenshot below). Moreover, the resulting map isn't working on the server. The server simply doesn't let me in, throwing an error.

How did @damienflament manage to use Goxel to create .vxl maps? Is there something I don't know?

[OverwhelmingFire]

Moreover Terravox cannot open this map too. Very disappointing

[guillaumechereau]

Hello @OverwhelmingFire. Yes the voxlap support is only as good as it gets, since I never used AoS. Can you give a bit more information? What error does the server give? Can you attach a test image that I can try?

[OverwhelmingFire]

Hello @OverwhelmingFire. Yes the voxlap support is only as good as it gets, since I never used AoS. Can you give a bit more information? What error does the server give? Can you attach a test image that I can try?

I created another .vxl map to explain in details (I have already deleted the first one...). It's very simple and is completely filled with blocks (512x512x64) despite just one small area in the middle with a single layer of blocks (which by the game is interpreted as water). The program I used to create the map is Voxed (just the one @guillaumechereau mentioned in his post). Maps created by this program run on the server without a problem. The size of this original map is 2.2 KB.

Then I opened it in Goxel, went to the Image tab and resized the "canvas":

When I saved it under a different name, and it's seen that the size of the map become 4.5 KB.

I opened both files in a hex editor to compare the structure, and it definitely differs, though I cannot understand how exactly since I don't know the structure of .vxl files.

This is a screenshot of how it looks like on the server (I'm staying at the edge of the central "pit":

This is a screenshot of the server logs when I'm trying to enter the Goxel map version: I try to connect via launcher, enter the localhost address, but when the game tries to download the map it suddenly shuts down without even showing me an error, while the server says I simply disconnected (sorry, in my first post I said that the server shows an error, but it isn't, it just shows that I disconnected). I tried twice and didn't manage to join.

Here I attach the maps: maps.zip

[OverwhelmingFire]

Hello @OverwhelmingFire. Yes the voxlap support is only as good as it gets, since I never used AoS. Can you give a bit more information? What error does the server give? Can you attach a test image that I can try?

Sorry, I miss-mentioned you with @damienflament. It's very late in my country, and the attention decreases in night

[poVoq]

Has anyone ever gotten this to work with OpenSpades?

Would be really nice to use Goxel as a level editor for it.

[bztsrc]

Hi,

I've checked the hexdumps of the .vxl files in the attachment above.

This is the one that works (saved by Terravox):

$ hexdump -C map.vxl | head -4
00000000  00 00 00 00 7c 7c 7c 80  00 00 00 00 7d 7d 7d 80  |....|||.....}}}.|
00000010  00 00 00 00 7e 7e 7e 80  00 00 00 00 7f 7f 7f 80  |....~~~.........|
00000020  00 00 00 00 80 80 80 80  00 00 00 00 81 81 81 80  |................|
00000030  00 00 00 00 82 82 82 80  00 00 00 00 83 83 83 80  |................|

And the one which doesn't (saved by Goxel):

$ hexdump -C mapGoxel.vxl | head -4
00000000  00 40 3f 00 00 00 3f 00  7c 7c 7c 80 00 00 00 ff  |.@?...?.|||.....|
00000010  00 00 00 ff 00 00 00 ff  00 00 00 ff 00 00 00 ff  |................|
*
00000100  00 00 00 ff 00 00 00 ff  00 00 3f 00 7d 7d 7d 80  |..........?.}}}.|

Considering the differences I'm pretty certain even though they are both called '.vxl', these two files are not the same file format (assuming both files are storing the same voxel image, which they should).

Check out:

1. the first data 7c 7c 7c 80 should be at offset 4, however it is at offset 8 in Goxel's version, suggesting Goxel saved a completely different header (or more precisely, there should be no 00 40 3f 00 header but there is).
2. the second data 7d 7d 7d 80 should be at offset 12, but it is at offset 268, with lots of 00 00 00 ff in between.
3. both packets are prefixed by 00 00 00 00 in the original file, but by 00 00 3f 00 in the Goxel saved version. This suggests that the encoding is different too (or the dimensions differ, but there's no header with the dimension, so impossible to tell).
4. finally, if different voxel images are stored in these files, then these files are not suitable for debugging in the first place.

I think the main problem here is, there are no magic bytes to identify the format, and it looks like there's no header which could store the dimensions either. So different tools can (and they do) save completely different files by the same extension. BTW, according to this, the .kv6 format for example has such magic (0x6c78764b) and header, so trying to use the KV6Importer or KV6Exporter for these format obviously won't work either.

I could fix this in Goxel using vxl.cpp in Terravox, but I'd suggest to use other format in OpenShades instead, something that has magic bytes, proper dimensions and saved correctly by Goxel in the first place (.kv6 maybe? Or .vox?).

And I'm also certain OpenShades' developers are more willing to accept PRs than guillaumechereau (who didn't do anything about this for more than 5 years), so you have much better chances there. Or just keep using Terravox, it looks like a FOSS Qt app to me, should have no issues.

Cheers, bzt

[bztsrc]

Then I opened it in Goxel, went to the Image tab and resized the "canvas":

Taking a closer look at Terravox's source, its .vxl format only supports 512 x 512 x 64 sized "canvas", nothing else is valid. This could be very well the reason for the incompatibility.

Cheers, bzt

[poVoq]

There is this C library what might be possible to include in Goxel for .vxl read and write support support: https://github.com/xtreme8000/libvxl/

Also exporting .kv6 files would be very useful for making OpenSpades models.

[poVoq]

Turns out this has been done already: https://github.com/xtreme8000/goxel/commit/71efe67bc58f373c2fed9d2770c56898b5061746 Just needs to be backported.

[bztsrc]

@poVoq: libvxl does not respect the original 512 x 512 x 64 limitations (it saves arbitrary volumes and has some heuristic to detect dimensions on load).

Digging deeper, one thing is certain, the example C code in the original Ace of Spades map spec (the same code that Goxel uses) doesn't work at all and is buggy as hell. It saves incorrect lengths, often overflows, and sometimes it does not store any color data after the lengths packet header (probably this was the reason for the client crash).

I've tried to fix the example code in the spec, and it is almost perfect now, here's a diff of Terravox saved map.vxl from above loaded and saved with Goxel and compared with the original:

42324c42324
< 000a5530  00 00 3e 00 7f 7f 7f 80  31 40 70 80 36 40 74 80  |..>[email protected]@t.|
---
> 000a5530  00 00 3f 00 7f 7f 7f 80  31 40 70 80 36 40 74 80  |[email protected]@t.|
42340,142295c42340,142476
< 000a5630  00 00 3e 00 7e 7e 7e 80  33 44 74 80 30 42 74 80  |..>.~~~.3Dt.0Bt.|
< 000a5640  31 43 75 80 36 41 75 80  37 40 70 80 34 40 74 80  |[email protected]@t.|
< 000a5650  35 40 75 80 32 47 77 80  33 44 75 80 30 42 70 80  |[email protected].|
<      ....plus a bunch of litter

As you can see, there's only one difference in one of the packet lengths at offset ca. 2.2M (3f instead of 3e), and some litter at the end of the file. Otherwise this version of Goxel at least can load back the vxl it saves...

There were several problems:

1. the map array was filled up incorrectly. I've completely rewritten export_as_vxl with a proper iterator
2. the edges were incorrectly assumed as solids, creating additional (and invalid, see below) RLE packets
3. writing the variable top_colors_end to file overflowed
4. it was possible that top_colors_len was zero, therefore no colors written into the file, resulting in invalid packets and therefore corrupt vxl files

TODO: figure out why there's a garbage at the end

diff --git a/src/formats/vxl.c b/src/formats/vxl.c
index c2a3ba7a..1b863182 100644
--- a/src/formats/vxl.c
+++ b/src/formats/vxl.c
@@ -1,6 +1,7 @@
 /* Goxel 3D voxels editor
  *
  * copyright (c) 2016 Guillaume Chereau <[email protected]>
+ * copyright (c) 2022 vxl export fixes (still not perfect) bzt <bztsrc@gitlab>
  *
  * Goxel is free software: you can redistribute it and/or modify it under the
  * terms of the GNU General Public License as published by the Free Software
@@ -90,8 +91,8 @@ static int vxl_import(image_t *image, const char *path)
 
             color = (uint32_t*)(v + 4);
             for (z = top_color_start; z <= top_color_end; z++) {
-                CHECK(z >= 0 && z < d);
-                swap_color(*color++, cube[AT(x, y, z)]);
+                if(z >= 0 && z < d)
+                    swap_color(*color++, cube[AT(x, y, z)]);
             }
 
             len_bottom = top_color_end - top_color_start + 1;
@@ -132,9 +133,11 @@ static int vxl_import(image_t *image, const char *path)
 static int is_surface(int x, int y, int z, uint8_t map[512][512][64])
 {
    if (map[x][y][z]==0) return 0;
+/*
    if (x == 0 || x == 511) return 1;
    if (y == 0 || y == 511) return 1;
    if (z == 0 || z == 63) return 1;
+*/
    if (x   >   0 && map[x-1][y][z]==0) return 1;
    if (x+1 < 512 && map[x+1][y][z]==0) return 1;
    if (y   >   0 && map[x][y-1][z]==0) return 1;
@@ -215,7 +218,7 @@ void write_map(const char *filename,
                 bottom_colors_end = k;
 
                 // now we're ready to write a span
-                top_colors_len    = top_colors_end    - top_colors_start;
+                top_colors_len    = top_colors_end    - top_colors_start + 1;
                 bottom_colors_len = bottom_colors_end - bottom_colors_start;
 
                 colors = top_colors_len + bottom_colors_len;
@@ -223,10 +226,10 @@ void write_map(const char *filename,
                 if (k == MAP_Z)
                     fputc(0,f); // last span
                 else
-                    fputc(colors+1, f);
+                    fputc(colors, f);
 
                 fputc(top_colors_start, f);
-                fputc(top_colors_end-1, f);
+                fputc(top_colors_end, f);
                 fputc(air_start, f);
 
                 for (z=0; z < top_colors_len; ++z)
@@ -244,23 +247,26 @@ static int export_as_vxl(const image_t *image, const char *path)
     uint8_t (*map)[512][512][64];
     uint32_t (*color)[512][512][64];
     const mesh_t *mesh = goxel_get_layers_mesh(image);
-    mesh_iterator_t iter = {0};
     uint8_t c[4];
-    int x, y, z, pos[3];
+    int x, y, z, pos[3], bbox[2][3];
     assert(path);
 
+    if(!mesh_get_bbox(mesh, bbox, true))
+        return -1;
+
     map = calloc(1, sizeof(*map));
     color = calloc(1, sizeof(*color));
-    for (z = 0; z < 64; z++)
-    for (y = 0; y < 512; y++)
-    for (x = 0; x < 512; x++) {
-        pos[0] = 256 - x;
-        pos[1] = y - 256;
-        pos[2] = 31 - z;
-        mesh_get_at(mesh, &iter, pos, c);
-        if (c[3] <= 127) continue;
-        (*map)[x][y][z] = 1;
-        memcpy(&((*color)[x][y][z]), c, 4);
+
+    mesh_iterator_t it = mesh_get_iterator(mesh, MESH_ITER_SKIP_EMPTY);
+    while(mesh_iter(&it, pos)) {
+        mesh_get_at(mesh, &it, pos, c);
+        x = bbox[1][0] - 1 - pos[0];
+        y = pos[1] - bbox[0][1];
+        z = bbox[1][2] - 1 - pos[2];
+        if(x >= 0 && x < 512 && y >= 0 && y < 512 && z >= 0 && z < 64 && c[3] > 0) {
+            (*map)[x][y][z] = 1;
+            memcpy(&((*color)[x][y][z]), c, 4);
+        }
     }
     write_map(path, *map, *color);
     free(map);

Cheers, bzt

[bztsrc]

Hi,

I've added a little patch to the libvxl Goxel version too, so no matter which one @guillaumechereau chooses to be merged, OpenSpades compatibility guaranteed. :-)

The map.vxl (from above) loaded and saved with this libvxl Goxel has the same size, but it looks like there's a little hiccup with the alpha channel (I haven't changed anything in libvxl, just used fixed dimensions). I don't know if this matters or not.

< 00000000  00 00 00 00 7c 7c 7c 80  00 00 00 00 7d 7d 7d 80  |....|||.....}}}.|
< 00000010  00 00 00 00 7e 7e 7e 80  00 00 00 00 7f 7f 7f 80  |....~~~.........|
< 00000020  00 00 00 00 80 80 80 80  00 00 00 00 81 81 81 80  |................|
< 00000030  00 00 00 00 82 82 82 80  00 00 00 00 83 83 83 80  |................|
< 00000040  00 00 00 00 84 84 84 80  00 00 00 00 85 85 85 80  |................|
< 00000050  00 00 00 00 86 86 86 80  00 00 00 00 87 87 87 80  |................|
< 00000060  00 00 00 00 88 88 88 80  00 00 00 00 89 89 89 80  |................|
< 00000070  00 00 00 00 8a 8a 8a 80  00 00 00 00 8b 8b 8b 80  |................|
< 00000080  00 00 00 00 7c 7c 7c 80  00 00 00 00 7d 7d 7d 80  |....|||.....}}}.|
< 00000090  00 00 00 00 7e 7e 7e 80  00 00 00 00 7f 7f 7f 80  |....~~~.........|
< ...lot more lines
---
> 00000000  00 00 00 00 7c 7c 7c 7f  00 00 00 00 7d 7d 7d 7f  |....|||.....}}}.|
> 00000010  00 00 00 00 7e 7e 7e 7f  00 00 00 00 7f 7f 7f 7f  |....~~~.........|
> 00000020  00 00 00 00 80 80 80 7f  00 00 00 00 81 81 81 7f  |................|
> 00000030  00 00 00 00 82 82 82 7f  00 00 00 00 83 83 83 7f  |................|
> 00000040  00 00 00 00 84 84 84 7f  00 00 00 00 85 85 85 7f  |................|
> 00000050  00 00 00 00 86 86 86 7f  00 00 00 00 87 87 87 7f  |................|
> 00000060  00 00 00 00 88 88 88 7f  00 00 00 00 89 89 89 7f  |................|
> 00000070  00 00 00 00 8a 8a 8a 7f  00 00 00 00 8b 8b 8b 7f  |................|
> 00000080  00 00 00 00 7c 7c 7c 7f  00 00 00 00 7d 7d 7d 7f  |....|||.....}}}.|
> 00000090  00 00 00 00 7e 7e 7e 7f  00 00 00 00 7f 7f 7f 7f  |....~~~.........|
> ...lot more lines

Cheers, bzt

Ps: it was a lot easier to add fixed sized maps with libvxl than to figure out why does the original AoS example code add extra bytes to the file...

[guillaumechereau]

Closing this because I think this should work now. Feel free to reopen if needed.