new: added support for xyz-files on import

[Sythelux]

I have xyz-Files in my project which look roughly like this:

324000.00 5682000.00 157.71 324001.00 5682000.00 157.70 324002.00 5682000.00 157.66 324003.00 5682000.00 157.64 324004.00 5682000.00 157.66 324005.00 5682000.00 157.68 324006.00 5682000.00 157.65 324007.00 5682000.00 157.64 ...

the first line is base X and Y. The rest builds on top. The File is 4,000,000 lines long so the square root is 2000 and therefore the region is 2000x2000 big.

I don't like the crop so it is rounding to the next 2048+1 and fills 2000 - 2048 with z level 0. (I actually would like to have, what the hole tool is doing there, but I don't know how to achieve that)

[Calinou]

Is this format standardized anywhere? Also, how do other engines handle text-based heightmap file formats?

Using this format should be documented somewhere. Otherwise, the likelihood that anyone uses this feature in the long term is rather low.

[Sythelux]

Ther format is used by ARCGIS or more general in GIS realm:

https://www.esri.com/arcgis-blog/products/arcgis-online/3d-gis/creating-and-using-elevation-layers-in-arcgis-online-arcgis-pro/?rmedium=redirect&rsource=blogs.esri.com/esri/arcgis/2016/08/30/creating-and-using-elevation-layers-in-arcgis-online

for some reason it is not very well documented, but if I download gis data from my state: https://www.geodaten.sachsen.de/ they just provide them.

(if I find a better source I provide it here)

Blender can officially read them, though it crashes when it tries to load the file I have.

I think the chance, that others can use it is quite high, as it is basically implemented as a csv with space as separator and i think it would be a good way to provide others with a text based file format for heightmaps

[Sythelux]

I was hoping to find a IEEE spec or something, but I can't instead I can only present the links below.

XYZ is a common Point Cloud Format together with LAZ as stated here:

https://blog.topodot.com/point-cloud-data-format-file-types/

There are multiple tools, where you can convert XYZ to and from LAZ

https://geozoneblog.wordpress.com/2021/06/10/lidar-in-xyz/

these formats are important for lidar scans of surfaces and are commonly used, when scanning surfaces of the earth.

as explained here: https://www.bezreg-koeln.nrw.de/brk_internet/geobasis/hoehenmodelle/index.html (apologies for German, but look at the pictures)

I have to work with those Data as I'm trying to make a Digital Twin of a City in Germany and they only provide me with either LAZ or XYZ height maps, as they are more common than EXR or PNG

[Zylann]

Overall if this is so common in GIS I dont see why it should not be supported here, but the format can be wildly impractical to convert because points could be anywhere, so there are likely assumptions to make here. I'm surprised GIS software is exporting in such format, it looks highly inefficient when it comes to representing heights with an image, even single-channel PPM would be less data.

[Sythelux]

Overall if this is so common in GIS I don't see why it should not be supported here, but the format can be wildly impractical to convert because points could be anywhere, so there are likely assumptions to make here. I'm surprised GIS software is exporting in such format, it looks highly inefficient when it comes to representing heights with an image, even single-channel PPM would be less data.

I agree it is highly inefficient. Sadly I have those issues on other places, too e.g. CityGML dataformat is based on XML, but can't be parsed with a DOM-Parser, because it has inconsistencies with dtd headers. (sorry for half way off-topic). I am provided with these Files and have no other choice and I am just hoping, that someone who has to do it again, might find this implementation and can use it.

That is, when I fixed the obvious issues, which you outlined.

I'll try to change the check function now to iterate through the whole file and calculate the x-extend and y-extend and remove the dependence on the file.size/line.length.

If you know of a memory efficient way to do that in GDScript, let me know.

[Zylann]

I was thinking actually to give a try myself at an implementation, but for this I would need to know what the convention would be to read such files. From what you wrote I assume that:

• The first point is the origin, which is on top-right if we were to see from satellite view. Actually I believe Z is the one that's flipped, if we consider X to be "to the right"? Which would align with the fact that in the OpenGL convention, if you take a camera forward direction in Godot, it will look towards negative Z. Or will that not result in the right result still?
• The first point's X and Y are not necessarily 0
• Next points are placed on a grid that goes to the left on each point, and when the row is complete, goes back to the origin X coordinate, goes down 1m and continues
• The grid of points have their X and Y coordinates separated by 1 unit exactly (whatever that unit is)
• Heights use the same unit as X and Y

When it comes to flipping, perhaps I would not bother doing it in GDScript, it can always be done at the end with Image.flip_x(). And for pre-filling with 0, that can be done with Image.fill. Curious that you had noise instead.

On a side note, I wish Godot simply had streams like C#. That would make it so efficient to parse text in comparison, you'd just read floats one by one without bothering about strings, while get_line and get_csv_line are an allocation hell

[Zylann]

After doing some tests myself, Godot is reaaaally slow at reading a 2000^2 file. I can't even begin to imagine that for a 4000^2 dimension. That format really sucks... but I suspect the available API isn't helping either, performance-wise. doing operations in memory might help but for the biggest files that would require 1.6 Gb of free memory.

What I found so far:

• Using get_line() followed by split_floats is faster than get_csv_line()
• Using Vector3 or Vector2 to store variables is slower (and not recommended given how big X and Y values can be)
• Counting lines with get_8() in GDScript is wayyyy slower than using get_line(). I blame GDScript considering the implementation of get_line() uses get_8() too in C++, only it also does construct a string which is converted to UTF8 and then thrown away
• Image.fill() and Image.flip_x() take about 20ms, which is nothing compared to the rest

[Sythelux]

Thank you very, very much for working through this. I will change the code according to what you found as best as I can and add the commit.

(I'm feeling dumb for a few things. Like I could've used image.flip_x() instead of doing a stupid calculation, but didn't thought about that in the moment I was coding it.)

[Zylann]

Added in e51c544cd8282bae83ba4ef0fc10884e9216d37a, you can check if it works with your file

[Sythelux]

I merged it back into my fork and it works.

Thank you for taking care of it.