Make runtime loading compatible with threads ?

[Lertsenem]

Hi ! Maybe this is actually not an issue, but I have not been able to figure a solution yet.

So I'm trying to build a very (very) big map for the player to play in. To avoid gulping too much memory (and also to avoid in-editor freezes when opening Godot ^^"), I'm dividing the map in more manageable chunks of 300×300 tiles. In tiled I can then use the world feature to stitch those chunks together.

In Godot I built a Node which is able to read info from the world file, and load neighboring chunks when needed, using YATI runtime importer. It works, but with a minor issue : the import operation takes quite some time and freezes the main thread for about 2 seconds when loading a new chunk, which is very annoying from the player perspective.

To solve this issue, I tried using Threads thusly :

var t = Thread.new()
var c = load_map.bind(m)
t.start(c)

with load_map being :

func load_map(filename):
    var map_scn = Importer.import("res://maps/%s" % filename)
    map_scn.name     = filename
    map_scn.position = Vector2(map["x"], map["y"])
    maps_parent.add_child.call_deferred(map_scn.duplicate())

Unfortunately this approach does not work, and the entire game freezes with error such as :

E 0:00:22:0692   TilemapCreator.gd:273 @ handle_layer(): This function in this node (TileMap) can only be accessed from either the main thread or a thread group. Use call_deferred() instead.
E 0:00:22:0707   TilemapCreator.gd:598 @ create_map_from_data(): This function in this node (level) can only be accessed from either the main thread or a thread group. Use call_deferred() instead.
E 0:00:22:0707   TilemapCreator.gd:486 @ create_polygons_on_alternative_tiles(): This function in this node (level) can only be accessed from either the main thread or a thread group. Use call_deferred() instead.

(of course I tried using call_deferred() as suggested, to no avail).

Now my understanding is that these errors mean the runtime import is not Thread safe, and so I should not use it with Threads. But now I'm all out of ideas on how to tackle my problem, which is why I'm coming here. :]

• Is there a way to make the runtime import Thread-safe?
• Is there another preferred solution I'm not seeing for this kind of issue?

[Kiamo2]

• Is there a way to make the runtime import Thread-safe?

Unfortunately that's beyond my restricted Godot knowledge (others to the rescue...?)

• Is there another preferred solution I'm not seeing for this kind of issue?

Same ...

[Kiamo2]

Are there any new findings here? How could we proceed with the issue?

[Lertsenem]

I'm not really good enough yet to know how to tackle this, sorry.

For now I'm living with the ~3s freezes each time I load a new map. I guess as a last resort I could maybe pre-convert the maps from Tiled to Godot nodes and save them as Godot Scenes, that should be faster to load. But I would lose the ability to make modifications in Tiled and have them automatically incorporated in game.

[Kiamo2]

As far as YATI is concerned, I doubt that it's possible to make the runtime thread-safe in the sense that anyone can use it without even wasting another thought on thread programming. Maybe I'm wrong, I haven't had to deal with threads in all my programming tasks yet.

A few excerpts from the Godot documentary I noticed: - Interacting with the active scene tree is NOT thread-safe. This one seems crucial to me. And they then recommend the use of mutex and semaphore - uuugh. Further: - ... only really useful if you have one thread loading data. - Only use more than one thread to generate scene data if you really know what you are doing...

[Lertsenem]

Further:

• ... only really useful if you have one thread loading data.

Yes, that's what I was trying to do and got some errors in return (see original post).

I don't really understand why, but I can't use YATI in a separate thread to have it load levels quietly in the background while the rest of the game is happening.

[Kiamo2]

Ok, I assumed you're trying to use multiple threads for loading your chunks, thus slightly missunderstood that. Anyway... Another idea - does the runtime import somehow interact with the active scene tree?

[Lertsenem]

Oh, it's very possible I'm doing that, indeed... It's been some time I touched that part of the code, I'll need to reread what I wrote. Thanks anyway!

[Kiamo2]

Any news here?