external uri references not handled

[rdiankov]

trafficstars

i see this pattern a lot in the code:

url = node.get('url')
if not url.startswith('#'): raise DaeMalformedError('Invalid url instance %s' % url)

which suggests that no one has thought about external references like "file:/my.dae#myid". How can we get support for external references? If file cannot be resolved, it would be great to at least store as a stub tell users this is an external reference.

[jterrace]

Yeah this is an oversight from the start, it doesn't deal at all with external references. We already have mechanism in the main Collada object that remembers whether the file was loaded from zip or from disk, and it has the ability to find external files with a given name, so we could do potentially support a few different URI types

[rdiankov]

at minimum, I was thinking of creating a self.instance_url variable for all urls that could not be resolved. that way when the object is saved back, it can set the instance_url string to the url attribute.

[jterrace]

Not sure I follow - what object would have the instance_url property?

[rdiankov]

it could be a python string or an object modeling the collada instance_* elements.

either way, external references are not something that i want my application to read, but i do want my application to be able to store them and then save them back to a dae when i'm done editing the other parts of the collada file. For example, imagine a scene dae that holds external references to the objects in scene and i want to add one more object.

[rdiankov]

"i want my application to read" as in i don't want it resolving into a file that it then loads into memory, that can take too much processing power

[jterrace]

Ah, I see. This is a tough problem. If you just want external references to be loaded, I think this is doable with some effort of work. If you want the external references to be weak references, or on-demand references, I think that will be more difficult.

Note that I don't think I've ever come across external references in a COLLADA file in a wild. I'm sure someone is using them, but I have yet to see one.

Also, if you want an example of a weak-type reference, that's how textures are loaded. When a collada.material.CImage is first loaded, it just keeps a reference to the path. If you try and call .data or .pilimage on the object, it tries to load the texture on-demand. This is nice for files with large textures if you never actually need to access the texture data. I could see something similar being done for things like instance_node, but some places it would be more difficult, like if you reference an external file for an input float array. What do you do if it's a weak reference? You won't have access to the float array and it's not clear how to recover from that without loading it.

[rdiankov]

that makes us the world's first users of external references with collada (yay!). unfortunately this is a feature that we cannot live without.

For now, how do we implement the low-hanging fruit like only elements that start with instance_*? Other external references like input don't have to be changed.

If you can show an example of one properly implemented instance_* element that holds an external reference stub that can be later used for saving, then i can copy the code across the other parts.

thanks!

[jterrace]

Can you submit a pull request with an example file that has an external reference in the collada/tests/data directory? This would be really helpful to work off, since I don't have any test files.

[rdiankov]

i added several new dae files with external references to my git clone. however, i can't do any new pull requests until you clear my current one. for the time being, you are welcome to look in my master

[jterrace]

Actually, pushing to your master branch just automatically updated the other pull request :) In general, it's best to work on features for pull requests in a separate branch for each so that you can work independently. For now, I can probably just cherry pick that commit though

[rdiankov]

FYI:

mug1external.dae references a simple one link object mug1.dae

lab3external.dae references an entire robot with multiple links and physics info

[rdiankov]

Given that elements like instance_physics_model can have children specific to the physics specification, it might be necessary to create a separate InstancePhysicsModel class to manage that info (assuming user doesn't want to dereference it)... what do you think?