Better URL support in wbt files

[omichel]

Problem with the current design

It is currently pretty complicated to understand which textures, protos, robot controllers, sounds or plugin are used by looking only at the content of a wbt file. This is because there are many search paths with complex priority rules. This defeats the principle of Uniform Resource Locator (URL).

This also yields to problems when exporting to X3D format for example, where it is difficult to refer to the correct resources used by the X3D file (see, textures name clash).

Proposed solution

We should take inspiration from the original VRML97 design principles where resources are located very precisely thanks to a better URL implementation.

1. The URL fields should be implemented as real URLs, either absolute or relative to the wbt/proto file:

• ImageTexture.url
• Mesh.url
• Background.*Url
• Camera.noiseMaskUrl
• ContactProperties.bumpSound
• ContactProperties.rollSound
• ContactProperties.slideSound
• Skin.model
• Robot.window [folder]
• Robot.controller [folder]
• Robot.remoteControl [folder]
• WorldInfo.physics [folder] Examples:
• url "textures/image.jpg"
• url "https://webots.cloud/R2021a/textures/webots_billboard.jpg"
• url "../textures/protos/textures/card.jpg"

2. Each PROTO used in a wbt/proto file should be declared at the beginning of the file using the simplified EXTERNPROTO syntax:

• EXTERNPROTO "../protos/MyProto.proto"
• EXTERNPROTO "https://webots.cloud/R2021a/robots/boston_dynamics/spot/protos/Spot.proto"
• EXTERNPROTO "https://mywebsite.com/myfolder/MyBot.proto" All the resources (images and subsequent EXTERNPROTO will be downloaded as cascading URLs). Icon URLs will be forged automatically by Webots to download the corresponding PROTO icons.

3. For URLs referring to a folder (marked [folder] in the above list), one should consider they refer to the folder containing the files to be used by Webots. Examples:

• controller "../controllers/my_controller/" will launch my_controller[.exe] located inside the my_controller folder.
• controller "../controllers/my_controller/my_controller.py" will launch my_controller.py with the default Python interpreter.
• "../plugins/physics/my_physics/" will launch my_physics.dll (or .dylib or .so depending on the platform).
• "../plugins/robot_windows/my_robot_window/" will use my_robot_window.html and my_robot_window.js from this folder All the dependencies (if any) of the should be in the same folder (or sub-folder) as the code URL. In case of a local URL, nothing need to be done. In case of a remote URL for a controller, remote control or physics plugin, Webots should download the .index file containing the complete folder listing (not including sub-folders): "controller "https://mysite.com/my_controller/my_controller.py" → Webots will download "https://mysite.com/.index", parse it and download the files and content of folders listed in there. The format for the .index file is the following:

Makefile
my_controller.c
my_data_folder/

Then, the my_data_folder should contain also a .index file that will list its content:

my_data.txt

The .index files should be generated and updated automatically by Webots when accessing these folders.

4. An official web site https://webots.cloud will host the official and contributed PROTO files and dependencies (images, meshes, etc.) in a way similar to that.

Webots will implement a cache for these resources so that for example https://webots.cloud/R2021a/robots/boston_dynamics/spot/protos/Spot.proto and its subsequent resources are downloaded only once and stored for subsequent use in WEBOTS_HOME/cache/https/webots.cloud/R2021a/robots/boston_dynamics/spot/protos/Spot.proto`, etc.

Benefits of the proposed solution

We will be able to remove any constraint on the webots project directory structure: a world file could be anywhere, including at root level and it will refer to its resources in a very explicit way. Such resources could be organized in any directory structure at the users' choice.

Webots will be more efficient at loading the resources of world files, as it will not have to explore a priority ordered list of folders, but it will load directly the file explicitly referred to by the wbt/proto file. Such a list may be very long in case of protos resources for example.

It will be simpler for the user to understand which resource file is actually loaded by Webots.

We won't need to distribute the Webots resources (textures, protos, controllers, plugins, etc.) in the Webots installation package. Instead, they will be downloaded automatically as the user needs them. This should make the installation package much lighter.

It will be convenient to use resources created by others using EXTERNPROTO with absolute URLs for example.

It will be easier to deploy Webots simulations in the cloud.

[LeoJhonSong]

That would be really cool!

[omichel]

1, 2 and 4 are implemented in Webots R2022b.

[BenjaminDeleze]

Do we want to implement point 3 in the future or can we close this?

[omichel]

Yes, I believe that implementing point 3 would make the world file format more consistent. So, let's keep it open.