Universalize UID support in all resource types

[reduz]

Ensures all resource types support UIDs in a project. This is required to fix:

• Scripts and many other resource types can´t be referenced by UID, only by path. If used in text (in a script) when refactored the references are lost.
• Path export properties can´t use UID for unsupported types and must resort to paths. When refactored this is always lost (the editor can´t feasibly get into this data to change it).
• Refactoring problems when files are moved outside the editor (this PR effectively fixes it).
• Editor properly refresh paths if they changed externally while opened (as example, git update). This needs to be addressed in a subsequent PR, but this one effectively sets the prerequisites.
• Slow editor refactoring times (as file paths need to be modified all across the board every time something is renamed). This PR allows always using UIDs, so refactoring paths file by file is no longer needed.

Resource types that do not support UID will get a .uid file appended to them (this includes .gd , .gdshader, .gdextension, etc files).

SCREENSHOT:

2D Platformer demo with uids auto generated:

(Note RESOURCE.md, these are considered text resources by Godot and exported, so the .uid is actually correct.)

TODO:

• ~~Editor refactoring not yet supported.~~
• ~~De-duplication (if two UIDs are the same) not yet supported.~~ FAQ:

Why not using metafiles instead to make it more generic?

1. There has not been any need for metafiles. Most resource types contain metadata on their own.
2. The .uid files are only used by the editor and are effectively gone on export, as this information is moved to the uid database. As such they are unsuitable as metafiles.

[Jordyfel]

As someone who spent some time working on the refactoring problems and thinking about the UID system I fully support this approach.

This will need some code in the filesystem dock to move the .uid files along with the main file, much like with .import.

To make sure, refactoring paths file by file will no longer be needed, means that users should be encouraged to hardcode uids instead of paths, right?

What do you mean by "editor refactoring not yet supported"?

Significant issue that will not be fixed by this: #95637

This comes from the problem that paths for extraction from imported 3d files, stored as import options, currently do not support uids at all, nor do they work with the refactoring logic in the filesystem dock. I've though about this a bit but am unsure (maybe store uid as a hidden import option?), but it seems like paths in .import files will need some significant effort to be made to fully work with refactoring.

[lyuma]

it seems like paths in .import files will need some significant effort to be made to fully work with refactoring.

Agree that this is important: hardcoded paths is one of the biggest issues with the advanced importer, especially given that it causes failed imports or even outright engine crashes in some cases if the dependent files are missing.

Allowing .import files to reference by UID is blocking some workflow improvements such as #86430 so it would be good to get this fixed in 4.4

[reduz]

@lyuma this seems to be a separate issue that will need a separate fix

[JoNax97]

This is a great step forward, but I think your assessment about meta files is a bit disingenuous.

Import files already contain editor-relevant metadata, and if in the future there's a need for saving anything else about code files, then you'll be in a situation where you either have to add another secondary file, rename the uid extension, or have a disconnect between the file intended type and it's content.

I think it would be preferable to future proof a bit here, and allow for meta files that are general and flexible enough to avoid needing more as hoc solutions going forward.

[reduz]

@JoNax97 If you ask me and I could redesign the engine from scratch, I would just have used generic metadata files for everything. Because that's not possible and the other formats already contain their own stuff, and their own metadata, this is the simplest solution for that.

[JoNax97]

I understand your point. Maybe this could be the first step in a progressive migration path.

[reduz]

@JoNax97 AFAIK that is probably Godot 5 material, so not likely to happen for a long time. We need to commit to a stable engine before major changes like this happening again.

[arkology]

Did I understand correctly that it will be illegal to add uid files to gitignore?

[reduz]

@arkology I would not do it :laughing:

[wstumpfz]

I too would like to vote to make this more like a generic metadata file. Currently I'm trying to add "pack file tags" to assets to facilitate the pack file exporting workflow in godot (https://github.com/godotengine/godot-proposals/issues/10580). The initial roadblock I've encountered is there's no real way to add metadata to PackedScenes [.tscn files] and GDScripts. As a proof of concept I added the ability for PackedScenes and GDScripts to have .import files to make it possible to save custom metadata for those assets. I had to edit some logic around how godot handles .import files, but it works. Having dedicated metadata files could be a better alternative, so it would be great though if you could expand the scope of this effort to make it a little more generic. It would make it significantly more futureproof.

[lyuma]

@wstumpfz

there's no real way to add metadata to PackedScenes [.tscn files] and GDScripts

The Godot philosophy is that the solution should follow the problem. I think a more palatable approach (less likely to break assumptions) would be to add a place in the .tscn format for metadata to be written. That would solve that specific sub-problem

scripts like .gd and .gdshader are a bit more complicated, because the code is human-readable, but you could imagine attaching something bulky like a texture into the metadata fields. Imagine for example, a plugin that assigns icons to scripts in metadata. If this image were dumped in base64 format into an annotation like @meta("foo", "Resource(bar...)") that could be unexpected.

If you limit script metadata to resource references and text strings, it could still be feasible to add annotations to scripts, but it wouldn't generalizable to 100% of cases, so this could be unexpected.

There's actually a funky example in Unity which is similar to what you describe. Unity shaders can be in text format like in Godot. However, there is a feature where a default texture can be assigned for a given texture slot. Unity actually puts that texture reference into the .shader.meta file. In Godot, it would not be feasible to put a texture into the body of a shader, and I can see the appeal to having a .meta file to store this type of info, as in these other engines.

My in-depth explanation of the problem with using .import for native formats (copied from RocketChat)

I think changing the definition of .import in this way is going to break a lot of assumptions, so you can do this in your own project, but I would consider it unlikely that it will be approved to merge into Godot what you are asking for is something like generic meta files,

I would read the discussion on the PR I linked, because it basically explains the problem with moving towards generic meta files. In fact, Reduz would be in favor of this type of change, but only for a major version

Sorry I didn't do a good job of explaining the issue with .import - basically, the way Godot thinks of things, assets are natively in resource format (.res / .scn). The purpose of the importer is to take a file in another non-native format and convert it to a native godot resource format, "remapped" to a native resource in .godot/imported

In other words, through this process, there are now two files at the same logical path in the project: there is the original file (such as a .fbx), and there is the .res in .godot/imported that it has been remapped to. For this reason, the original .fbx is read-only (you wouldn't want to overwrite a source file with a godot native resource in a different format.)

Now, separately, there is something called ResourceFormatLoader, which is what allows Godot to read .tscn, .res, .gdshader and so on, without a .import file. In essence, having a ResourceFormatLoader is what makes a format "native".

I think it is a very valid question to ask why loaders and importers are different types of things, but the fact is that they are. I think one answer is that loaders must be fast, since you will go through them every time you open a file, while importers usually are slow and therefore must cache a second native resource in .godot/imported.

For example, in Unity, every asset has a .meta file, and the meta file is the same format for every file type: even native assets have an importer, but it is set to NativeFormatImporter. (Though I wouldn't be surprised if it works similarly to Godot in the sense of reading the file directly and bypassing the import system)

So to circle back to your proposal, the problem here is that we are taking something with a resource loader, and treating it as if it is imported, but somehow without a resource in .godot/imported.

[huwpascoe]

Because that's not possible and the other formats already contain their own stuff, and their own metadata, this is the simplest solution for that.

A future PR will inevitably need to store more data, the simplest solution will add another string after the UID, and eventually the Godot metadata format will become ill-defined text files with .uid extension.

I'm calling it now. 🙂

[akien-mga]

It's easy enough to rename the extension in the future if we need, and have some compatibility code to handle the transition gracefully (keeping support for reading the old format, and adding support for converting to the new format). So there's no need to future proof and bikeshed now.

[reduz]

@huwpascoe Please check the FAQ I wrote, metadata does not make sense because:

• This is only for very few file types, not all.
• These files do not get exported.

If at some point you want to have metadata, it will have to be done in a different way.

[huwpascoe]

That's okay, Akien's response was convincing enough. I probably should've replied with more than an emote!

I do have a question though, sometimes assets are marked as don't import and show up as a little x icon, or maybe they'd be set for importing again later. Does the .uid handle this edge case?

[reduz]

@huwpascoe I think it should be fine, since these is in practice for assets that are not imported

[reduz]

Changes made:

• Proper uid:// syntax saved in uid file for more consistency.
• Unexposed the _has_custom_uid_support and just checked if virtual has been overriden, as suggested above.
• Moving, renaming, duplicating now supported in filesystem dock.

This PR is now ready for review!

[Saul2022]

• Moving, renaming, duplicating now supported in filesystem dock.

Does that mean it no longer slow af to rename files when there’s a lot of files in the project?( 10k, 20k)

[RevoluPowered]

@reduz Would it be worth adding editor functionality to resolve UID issues like this too? editor/editor_file_system.cpp:1259 - UID duplicate detected between res://textures/special/clip.png and res://textures/shaders/tangent-test.png.

I have a project that I created in 4.3.dev 3 exhibiting this problem with UID's I think the function might be generating duplicates higher than it should be.

[reduz]

@RevoluPowered Do you have an open issue about this? I feel that is unrelated.

[RevoluPowered]

@RevoluPowered Do you have an open issue about this? I feel that is unrelated.

OK I did this thanks! https://github.com/godotengine/godot/issues/97916

[KoBeWi]

I assume this is not intended? Seems like all non-Resource text files are getting this.

[reduz]

@KoBeWi it looks like its a resource file (someone must have added .cfg as a text file format).

[KoBeWi]

The same happens to txt and md files. They are displayed in FileSystemDock, but they aren't true Resources (they don't get included on export). Might be related to TextFile resource, but it's just a hack to allow displaying and editing text files.

[KoBeWi]

.gdshader files don't have uid.

[reduz]

@KoBeWi weird, they definitely should, will have to check

[reduz]

Added the suggestions. Text files no longer detected @KoBeWi gdshader seems to work for me:

[lyuma]

I didn't realize this applies to .gd scripts...

I'm not extremely thrilled about the prospect of having a ton of .uid files floating around my script directories, especially since they might be autogenerated by users of my addons. In particular, when I use an ide like vscode to write scripts, I would worry about uids getting mixed up when refactoring code

Is there still a plan to add a uid comment or annotation to scripts?

[KoBeWi]

The possibility is still there. This PR just adds a generic system that can be used by any Resource with no proper UID support.

[lyuma]

Sounds great, just wanted to make sure that possibility is still on the table.