Loading extra data via per task instructions fails

[Jorieke]

Describe the bug When adding the following in the per task instructions as project manager: This task involves loading extra data. Click [here] (https://hotosm.osmsharp.com/data/{x]/{y}/{z}/Baidoa_camp_mapping.osm) to load the data into JOSM. It should load an extra data layer for the concerned task area when opening the task. It however does not load any data. And you actually also cannot unselect your task anymore and pick another task or see the instructions.

I'm working in Edge on my computer usually, but have tried as well Chrome. I am trying it in this project https://tasks.hotosm.org/projects/14877 This is the link where the data is hosted that I want to use in the per task instructions: https://hotosm.osmsharp.com/data/{x]/{y}/{z}/Baidoa_camp_mapping.osm

Additional context See these comments on Slack: https://hotosm.slack.com/archives/C319P09PB/p1685519957147629

[Aadesh-Baral]

Hi @Jorieke The URL you've provided has a typo. The curly braces enclosing x is not closed as ] is used instead of }. Due to this one of the Tasking Manager endpoint was failing which resulted in error while trying to lock the task. Ideally there should not be any error even if the curly brace is not closed so this issue has been fixed in #5940. But you'll still need to fix the URL to load the data layer in JOSM.

Old URL: https://hotosm.osmsharp.com/data/{x]/{y}/{z}/Baidoa_camp_mapping.osm

Corrected URL: https://hotosm.osmsharp.com/data/{x}/{y}/{z}/Baidoa_camp_mapping.osm

I've also updated the per task instruction of project 14877 so you can also try to load the data into JOSM layer using specific task instruction.

[Jorieke]

@Aadesh-Baral apologies for the late reply, thanks a lot for the follow up! To get it to work I had to add in my link as well the JOSM remote control functionality. Just sharing for reference the link that is now working: http://127.0.0.1:8111/import?new_layer=true&url=https://hotosm.osmsharp.com/data/{x}/{y}/{z}/Baidoa_camp_mapping.osm

[Aadesh-Baral]

Is everything working as expected? or Do you want any enhancements to it?

[Jorieke]

@Aadesh-Baral thanks for checking! Well there is maybe one thing: with the link above it opens the data in a seperate layer. If I remove the new_layer=true part it will open in the same layer. It is however than not possible to upload the OSM data layer, despite having edited it in JOSM. I think I understand the reason for that, but it would be still more easy for us the have it opening in the same layer and than being able to upload it.

[Aadesh-Baral]

The per task instructions field's text is simply filled out by TM with the values for x, y, and z. So this cannot be fixed in the TM.

@tsmock Are there any JOSM url-side options available to fix this case?

[tsmock]

It would help if I had access to the generated osm file from osmsharp.com.

With that said, I would check and see if the <osm> xml stanza has upload='never' set first. If that is the case, when it gets merged into the active OSM layer, it blocks that layer from being uploaded. The more restrictive upload policy "wins".

[Aadesh-Baral]

@tsmock THere is the osm file generated from osmsharp.com for project 14877.

osm_sharp.zip

[tsmock]

That file seems to be "OK".

With that said, curl https://hotosm.osmsharp.com/data/64409/81419/17/Baidoa_camp_mapping.osm is showing something a bit different in the header.

- <osm version='0.6' generator='JOSM'>
+ <osm version="0.6" generator="OsmSharp" upload="never">

Importing the data from osmsharp into the OSM data layer will always make upload impossible (example: curl http://127.0.0.1:8111/import?url=https://hotosm.osmsharp.com/data/64409/81419/17/Baidoa_camp_mapping.osm). This is most likely a server configuration problem. If you do not have the ability to change server-side configuration to remove the upload="never" stanza, you will have to use the new_layer=true parameter.

tile_17_81419_64409_Baidoa_camp_mapping.osm.zip

As an alternative, I can modify the MapWithAI plugin to better support this specific type of external data.

This should be fairly easy, since I added support to the MapWithAI plugin for external data sources via remote control. For example, http://127.0.0.1:8111/mapwithai?url=https://hotosm.osmsharp.com/data/{x}/{y}/{z}/Baidoa_camp_mapping.osm

I'll note that (right now), the MapWithAI plugin doesn't replace {x}/{y}/{z}, so users will be getting all the data, or none of the data (depends upon the server). It also assumes that the server will ignore parameters it doesn't understand (not the case for osmsharp, apparently -- I send a crop_bbox={bbox} to the server if a tasking manager boundary layer is detected).

tile_17_81419_64409_Baidoa_camp_mapping.osm.zip

[Jorieke]

@tsmock I do not fully understand all your points, but it would be really great to be able to feed our data as an extra layer via the MapWithAi plugin. Do I understand that this is being planned?

[tsmock]

If you copy and paste the following url into your web browser with JOSM running (with MapWithAI installed and remote control enabled), you should get data from your data source, assuming you have downloaded the task area.

http://127.0.0.1:8111/mapwithai?url=https://hotosm.osmsharp.com/data/%7Bx%7D/%7By%7D/%7Bz%7D/Baidoa_camp_mapping.osm

Note: It is important that you use "url encoded" { and } (%7B and %7D respectively) since otherwise the { and } characters get removed due to url parsing, and the placeholders {x} become x, which are not replaced.

If you are using this in a TM task, I believe {x}/{y}/{z} get replaced, and you should just use the raw { and } characters.

I made some code changes after my previous comment to support {x}/{y}/{z}` parameters.

[Jorieke]

Hello @tsmock

I'm only now properly trying it out again, apologies for that.

I am trying to do as you say (opening link in browser, JOSM running, MapWithAI installed & remote control enabled) with this URL:
http://127.0.0.1:8111/mapwithai?url=https://hotosm.osmsharp.com/data/{x}/{y}/{z}/Baidoa_camp_mapping.osm

There is however no data opening and I'm getting the following error:

Any chance you could help me out again? It would be so great if this could work!

[tsmock]

It worked for me. Have you updated JOSM and the plugin recently?

[swise5]

Hi @tsmock - a bit of a weird case: on an older version of mapwithai (1.7.1.6) I'm able to load a new layer with mapwithai from a Tasking Manager setup using

http://127.0.0.1:8111/mapwithai?url=https://hotosm.osmsharp.com/data/{x}/{y}/{z}/Baidoa_camp_mapping.osm

On my JOSM, this was producing (for example)

GET https://hotosm.osmsharp.com/data/325681/266650/19/Baidoa_camp_mapping.osm (returning a 200)

However, the users of the latest release - 823 - keep getting various errors. A group of us tried debugging this problem on a mid-month zoom, and one members (peculiar_theatre) noticed that he was getting a 404 with the following call:

GET https://hotosm.osmsharp.com/data/325684/266654/19/Baidoa_camp_mapping.osm&crop_bbox=43.629456,3.095265,43.630142,3.09595

So the parameter crop_bbox is causing trouble. peculiar_theatre suggested that we could circumvent the strangely added ampersand/crop by adding a question mark (as %3f), so we're now using the following as a workaround:

http://127.0.0.1:8111/mapwithai?url=https://hotosm.osmsharp.com/data/{x}/{y}/{z}/Baidoa_camp_mapping.osm%3f

In general, he suggests that using a library to handle URLs might be better - the assumption that the url will already contain a parameter isn't a safe one, and is likely to cause silent failures like this.

[tsmock]

At the time the original code was written, it was a safe assumption -- it was only the FB road data at the time. Anyway, I am stuck with the libraries that JOSM includes by default (there are very few of those).

I'll have a fix out for that problem in an hour or two.

[Jorieke]

Thanks for looking into it together with me @swise5 and mapathon folks. @tsmock, fixed in an hour or two - wow! :D

[swise5]

Ah, as usual - it's a victim of its own usefulness :)

Thanks and I hope the fix goes smoothly!