Listing directories in exported project returns only `*.import` files

[suetca]

Godot version: Godot v3.0.6 Stable x64

OS/device including version: Windows 8.1

Issue description:

Shortly I tried to scan my res://audio/supersounds directory for every .wav file it had into array, and then play track by picking randomly with randi() % array.size(). But i've met problem where already exported project return empty array. So thinking about it might be problem with .wav, i've added some .png and .ogg into the folder and even removed file type check and it did some progress, it returned only .import type files. That's not what I wanted. In editor everything showed how expected.

Some guy in godot questions topic did reproduction and met the same problem. But he somehow got some .png and other types in output. Being surprised I also went and created empty project with same function to scan, but added custom .png into res:// and tried to list the thing. This is what've got:

[Node.gd.remap, Node.gdc, Node.tscn, airSpike.png.import, default_env.tres, icon.png, icon.png.import, project.binary]

Apparently default files and main scene are somewhat defended against it!?

After... I went already here to search for answer I thought it might be #25347, but i checked it's reproduction and it works fine, so file_exists() isn't the case.

Im at dead end now.

Steps to reproduce: Create new scene, add root node, add following script to it:

func _ready():
	print(list_files_in_directory("res://"))

func list_files_in_directory(path):
	var files = []
	var dir = Directory.new()
	dir.open(path)
	dir.list_dir_begin()
	while true:
		var file = dir.get_next()
		if file == "":
			break
		elif not file.begins_with("."):
			files.append(file)
	dir.list_dir_end()
	return files

Export, launch. Executable debug console should print something similar to this:

[Node.gd.remap, Node.gdc, Node.tscn, airSpike.png.import, default_env.tres, icon.png, icon.png.import, project.binary]

Expected:

[Node.gd.remap, Node.gdc, Node.tscn, airSpike.png, airSpike.png.import, default_env.tres, icon.png, icon.png.import, project.binary]

Minimal reproduction project: exported.zip project.zip

[bruvzg]

Original import sources are not preserved on export (except some special cases like icon.png), only imported stuff (converted to internal format) in the hidden .import folder is included.

You can still use load / preload these files using original path.

[suetca]

Original import sources are not preserved on export (except some special cases like icon.png), only imported stuff (converted to internal format) in the hidden .import folder is included.

You can still use load / preload these files using original path.

Ok, but i don't want to load any files. I need to do what i need to do i.e. get full list of files i have in my directory. Maybe there is another way, if yes then how?

[bruvzg]

Maybe there is another way, if yes then how?

1. Make sure Godot is not running.

2. Create empty .gdignore file in the sub-folder you want to preserve as is

3. Delete all *.import files from this folder if there are any

4. Add file extensions to "export non-resource files" in export options

Resulting PCK content:

Output: [airSpike.png]

[bruvzg]

Another option: you can try to use this contraption to directly create PCKs from folder without any changes to files or to extract exported PCK, add files and repack it back.

[suetca]

I simply can get why this does not works properly. If it works in editor it's intended to work everywhere, else this is either a huge issue or listing function is completely pointless to have in engine. I don't believe in workarounds cause they tend to fail everytime you add minor changes to anything :S

[tempdoom]

Would like to say that this solution fixed the issue of importing for me, now I've got files which export properly without a *.import, however now I'm getting a "no loader" error when I run the program, and I can't use any of the files while running in the editor.

[LinuxUserGD]

Another option is to use the whole project folder and put the executable inside.

[bruvzg]

properly without a *.import, however now I'm getting a "no loader" error

load/preload are looking for xxx.import and associated resources stored in .import hidden folder, it the same way you won't be able to use it to load arbitrary .png file outside your project. See https://docs.godotengine.org/en/latest/getting_started/workflow/assets/import_process.html for more info.

But you can load it manually (process is different for various file type: for images you should use Image, for sound files load it as file and set AudioStream*.data).

Won't work unless test.png is imported (no test.png in PCK, but test.png.import and .import/test.png.{hash}.stex instead):

	$TextureRect.texture = load("res://test.png")

This will work with any .png file, inside PCK (path with res:\\ prefix), or somewhere in the user filesystem (path with no prefix):

	var image = Image.new()
	image.load("res://test.png")
	var texture = ImageTexture.new()
	texture.create_from_image(image)
	$TextureRect.texture = texture

[bruvzg]

Related issues: #22433, #18367, #17848, #17748

[suetca]

Another option is to use the whole project folder and put the executable inside.

Isn't it a kind of absurd to distribute to other users access to all the sources of your game. Yes, the engine is open-source, but let's say you have something related to micro-transactions in your game, or your saving data is encrypthed and the key is hidden within your code. You have to think ahead and outside the box of using your game on your machine only.

[LinuxUserGD]

@timCopwell I have the source code of my game on Github, so it doesn't matter for me.

[bruvzg]

Another option is to use the whole project folder and put the executable inside.

You won't be able to load any changed/added files (or any files at all it you do not distribute .import folder as well).

Isn't it a kind of absurd to distribute to other users access to all the sources of your game.

Export is not much of protection. You can easily decompress PCK, you can convert scenes back to text versions and decompile gdscripts to the state almost identical to source. PCK does not store original files, but you still can restore it from .import folder stuff.

[tempdoom]

load/preload are looking for xxx.import and associated resources stored in .import hidden folder, it the same way you won't be able to use it to load arbitrary .png file outside your project. See https://docs.godotengine.org/en/latest/getting_started/workflow/assets/import_process.html for more info.

But you can load it manually (process is different for various file type: for images you should use Image, for sound files load it as file and set AudioStream*.data).

Won't work unless test.png is imported (no test.png in PCK, but test.png.import and .import/test.png.{hash}.stex instead):

	$TextureRect.texture = load("res://test.png")

This will work with any .png file, inside PCK (path with res:\\ prefix), or somewhere in the user filesystem (path with no prefix):

	var image = Image.new()
	image.load("res://test.png")
	var texture = ImageTexture.new()
	texture.create_from_image(image)
	$TextureRect.texture = texture

Is there an *.ogg/streamplayer equivalent to the code mentioned above?

[bruvzg]

Is there an *.ogg/streamplayer equivalent to the code mentioned above?

Something like this:

	var ogg_file = File.new()
	ogg_file.open("res://test.ogg", File.READ)
	var ogg_stream = AudioStreamOGGVorbis.new()
	ogg_stream.data = ogg_file.get_buffer(ogg_file.get_len())
	ogg_file.close()
	$StreamPlayer.stream = ogg_stream

[nathanfranke]

Unfortunately bruvzg's workaround has been fixed and no longer works in 3.2 beta 2

Edit: My new workaround is to have a file with the directory list inside of it. You can then use File and get_line()

[KoBeWi]

Ok, but i don't want to load any files. I need to do what i need to do i.e. get full list of files i have in my directory. Maybe there is another way, if yes then how?

If only .import files are listed, why not look for .wav.import instead of .wav? Every resource has its equivalent import file, so effectively it's the same.

[KoBeWi]

The fix for this issue would be documenting in get_files() method description that it lists the factual directory content and it will differ on export.

Then we could add a method named get_imported_files() (or similar) that returns the original files based on imports. It should likely also list .gd files based on .gdc, as it's a similar thing (see #42507).

[DaGamingWolf]

This seems unnecessarily confusing. I'm not sure why this has been a years long battle and yet no documentation under the load function mentions the fact that files will fail to load correctly after export if you don't jump through a bunch of extra hoops.

[vnen]

@DaGamingWolf there's no problem with load(), it works with the original path even if the file is technically not there. The difference is only if you list files in the project since the exported package contents may be different. This is mentioned in both FileAccess and DirAccess documentation pages.

[DaGamingWolf]

@DaGamingWolf there's no problem with load(), it works with the original path even if the file is technically not there. The difference is only if you list files in the project since the exported package contents may be different. This is mentioned in both FileAccess and DirAccess documentation pages.

that is only true on the specific condition that convert text resources to binary is set to false in the project settings. if it is set to true, the load() function is incapable of fetching the resource. That should be mentioned in the documentation if that is intended functionality. currently, there are no references to that specific setting in either class, nor its effects on the capability to use the load() function. This would have relieved a massive headache, and i would not have had to rely on the luck of seeing the setting and changing it to see if it affects the way resources are remapped such that the load() method would function.

[Jowan-Spooner]

What is the proposed way of making sure a specific resource file exists in an exported game? I'm working in godot 4.0.3 on a plugin. The plugin will save a list of files to add at runtime, but as files could always be removed we check first, to show a warning if the file has been moved or removed.

if FileAccess.file_exists(path:String):
    var resource = load(path)
    # other stuff
else:
    printerr("File "+path+" couldn't be accessed.")

This fails during export because file_exists() returns false. Do I manually have to check if .import, .remap have been added? It feels very weird that I can load a file that godot tells me is not there. I understand that FileAccess and DirAccess are supposed to represent the actual file system, but then what can I use use to list/test paths that are actually accessable (even if nonexistant)?

The only general purpose solution I can see is using

func is_valid_file(path:String) -> bool:
    return FileAccess.file_exists(path) or FileAccess.file_exists(path+".remap") or FileAccess.file_exists(path+".import")

Which idk. Seems pretty clunky for such a simple task.

[KoBeWi]

You can use ResourceLoader.exists()

[Jowan-Spooner]

Perfect! Thanks a lot!

[ePirat]

I just stumbled over this when I ran into https://github.com/godotengine/godot/issues/66014, for now I manually have to remove the .remap suffix to properly pass a path I get from DirectoryAccess to the ResourceLoader, this hardcoding of essentially implementations details seems not really ideal…

[AThousandShips]

@ePirat Did you mean to link another issue? You linked this exact issue.

[ePirat]

@AThousandShips Yes indeed, fixed.

[dalexeev]

See also:

• #37646
• #53339
• #56737

[rozab]

If we have ResourceLoader.exists(path) why can't we have ResourceLoader.get_files(path)?

It seems to me that the problem is it's currently difficult to do filesystem-y stuff with resources because whether or not the files actually exist at the paths they claim to is completely dependent on esoteric project configuration.

I want to do something similar to OP - I have a few hundred audio files and at runtime I just want to load them into AudioStreamRandomizers based on their filenames. I would expect this to be the main kind of thing people are doing with the filesystem, but it breaks on export.

Right now, the fixed version looks something like:

func get_resources(path):
	output = []
	for fname in DirAccess.get_files_at(path):
		if fname.ends_with(".import") and ResourceLoader.exists(fname.trim_suffix(".import")):
			output.append(fname.trim_suffix(".import"))
	return output

This is neither intuitive nor convenient. I think it would be best to supply another set of directory operations for those who don't actually want to deal with the concrete filesystem, just the tree of imported resources.

[tadeaspaule]

Hi everyone, I am currently struggling with this as well. I have mostly gotten things working, here is the code I'm using, maybe it will help you as well:

static func get_files(dirpath : String, must_ending: String = "") -> Array[Filename]:
  var dir_da = DirAccess.open(dirpath)
  if dir_da == null:
    return []
  var files : Array[Filename] = []
  for filename in dir_da.get_files():
    if filename == ".hidden": continue
    var p = dirpath + "/" + filename
    if p.ends_with(".import"): p = p.replace(".import","")
    elif p.ends_with(".r4") or p.ends_with(".json"): pass
    else: continue
    var fn = Filename.new(p)
    if must_ending != "" and fn.extension != must_ending:
      continue
    files.append(fn)
  files.sort_custom(func(a,b): return Helper._string_compare(a.base, b.base))
  return files

This uses the workaround from https://github.com/godotengine/godot/issues/14562 , notice how it is taking .import files instead of the normal ones (for example png files). Change the elif p.ends_with... line to suit whatever extensions that dont generate import files, that you want to keep (in my case, json and r4). Filename class is just

class_name Filename

var path
var base
var full
var extension

func _init(path : String):
  self.path = path
  var slash = path.rfind("/")
  var filename = path.substr(slash + 1) 
  base = filename
  full = filename
  extension = ""

  var dot = filename.rfind(".")
  if dot != -1:
    base = filename.substr(0, dot)
    extension = filename.substr(dot + 1)
  else:
    base = filename
    extension = ""

Then you just use that static func (I have it as Helper.static_func) instead of calls to get_files or get_files_at. Here are some examples:

  var spells_dir = DirAccess.open(spells_path)
  for class_n in spells_dir.get_directories():
    for namepair in Helper.get_files(spells_path + "/" + class_n):
      var texture = namepair.full
      var spell_id = "%s/%s" % [class_n, namepair.base]
      spells[spell_id] = ResourceLoader.load("%s/%s/%s" % [spells_path, class_n, texture]) as Texture2D

for m in DirAccess.get_directories_at(merchants_path):
    for mood in Helper.get_files("%s/%s" % [merchants_path, m]):
      merchants["%s/%s" % [m,mood.base]] = ResourceLoader.load("%s/%s/%s" % [merchants_path, m, mood.full]) as Texture2D

Two things you should do as well: just change all calls for imported assets (textures, models, etc) from load() into ResourceLoader.load, and all calls to FileAccess.file_exists to ResourceLoader.exists. careful with the find/replace though, as https://docs.godotengine.org/en/latest/tutorials/assets_pipeline/import_process.html says, ResourceLoader.load won't work for non-imported things (I'm assuming json for example, haven't tested)