Link of data source for milestone is out of date

[didier2020]

link source of open data is modified for milestone see https://www.data.gouv.fr/fr/datasets/bornage-du-reseau-routier-national/ and https://github.com/osm-fr/osmose-backend/pull/1461

[frodrigo]

The update cause an issue

COPY milestone_fr_metropole, line 2: "01/01/2022;01A903905CD;2;1;C;0;788;874126,39;6595769,32;0;D;1"

[frodrigo]

The new field separator is ";".

But the CSV content invalid Lambert 93 coordinates

Maybe oversea un-projected data.

[didier2020]

as it's metropole milestone there is a filter on field "depPr" line 68 => if len(row['depPr']) == 3: return False line 70 => csv line2 is filtered (?) because this is not a milestone (seek 'A9' in field route)

[didier2020]

tested with docker => error line 2 is decimal delimiter in for xy

• test suppress decimal => "01/01/2022;01A903905CD;2;1;C;0;788;874126;6595769;0;D;1" => ok
• test changed decimal => "01/01/2022;01A903905CD;2;1;C;0;788;874126.39;6595769.32;0;D;1" => nok

[didier2020]

tested ok with docker monaco see https://github.com/osm-fr/osmose-backend/pull/1488

[frodrigo]

On my side it fails with this patch

diff --git a/analysers/analyser_merge_milestone_FR_metropole.py b/analysers/analyser_merge_milestone_FR_metropole.py
index 0f24a3e6..3ea8c8cf 100644
--- a/analysers/analyser_merge_milestone_FR_metropole.py
+++ b/analysers/analyser_merge_milestone_FR_metropole.py
@@ -45,11 +45,19 @@ class Analyser_Merge_Milestone_FR_metropole(Analyser_Merge_Point):
             CSV(SourceDataGouv(
                 attribution="data.gouv.fr:Ministère de la Transition écologique et solidaire",
                 dataset="57a83c3dc751df5b90bb5dd5",
-                resource="8b3b36e8-e45a-48c0-8a70-3336d9769948")),
+                resource="8b3b36e8-e45a-48c0-8a70-3336d9769948"),
+                separator=';'),
             Load_XY("x", "y", srid = 2154,
                 xFunction = Load_XY.float_comma,
                 yFunction = Load_XY.float_comma,
-                where = lambda row: self.is_milestone(row)),
+                where = lambda row: (
+                    self.is_milestone(row) and
+                    row["x"] and row["y"] and
+#                                                                                                            653006.6
+                    float(row["x"].replace(",", ".")) > -378305.81 and float(row["x"].replace(",", ".")) < 6093283.21 and
+#                                                                                                           6867037.55
+                    float(row["y"].replace(",", ".")) > 1212610.74 and float(row["y"].replace(",", ".")) < 7186901.68
+                )), # Check for valid Lambert98 coords
             Conflate(
                 select = Select(
                     types = ["nodes"],

Error

2022-06-21 15:25:21   error: res=[['653006,6', '6867037,55'], '01/01/2022', 'A0001', '0', '93', 'N', '0', '0', '653006,6', '6867037,55', '0', 'D', '93']
2022-06-21 15:25:21   error: ret=None
2022-06-21 15:25:21   error: error on analyse merge_milestone_FR_metropole...
2022-06-21 15:25:21     Traceback (most recent call last):
2022-06-21 15:25:21       File "./osmose_run.py", line 263, in execc
2022-06-21 15:25:21         analyser_obj.analyser()
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Osmosis.py", line 193, in analyser
2022-06-21 15:25:21         self.analyser_osmosis_common()
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Merge.py", line 1345, in analyser_osmosis_common
2022-06-21 15:25:21         table = super().analyser_osmosis_common()
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Merge.py", line 1189, in analyser_osmosis_common
2022-06-21 15:25:21         table = self.load.run(self, self.parser, self.conflate, self.config.db_user, self.__class__.__name__.lower()[15:], self.analyser_version())
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Merge.py", line 951, in run
2022-06-21 15:25:21         return super().run(osmosis, parser, conflate, db_schema, default_table_base_name, version)
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Merge.py", line 857, in run
2022-06-21 15:25:21         osmosis.run0((sql01_ref if conflate.osmRef != "NULL" else sql01_geo).format(table = table, geom = self.geom, validationGeomSQL = self.validationGeomSQL, where = Select.where_attributes(self.select), distinct = distinct, order_by = order_by), insertOfficial)
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Osmosis.py", line 454, in run0
2022-06-21 15:25:21         self.run00(sql, callback)
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Osmosis.py", line 445, in run00
2022-06-21 15:25:21         ret = callback(res)
2022-06-21 15:25:21       File "/home/fred/osmose-backend/analysers/Analyser_Merge.py", line 845, in insertOfficial
2022-06-21 15:25:21         "fields": dict(zip(dict(res).keys(), map(lambda s: (s is None and None) or u'{0}'.format(s), dict(res).values()))),
2022-06-21 15:25:21       File "/home/fred/osmose-backend/venv/lib/python3.7/site-packages/psycopg2/extras.py", line 146, in execute
2022-06-21 15:25:21         return super().execute(query, vars)
2022-06-21 15:25:21     psycopg2.errors.InternalError_: transform: couldn't project point (653007 6.86704e+06 0): latitude or longitude exceeded limits (-14)

[didier2020]

it's not "my" code : https://github.com/didier2020/osmose-backend/commit/5c54ef164cada26e345b3f0962f10ebdc8a01f94

[frodrigo]

it's not "my" code : didier2020@5c54ef1

No. I made this previously. Because adding the ";" is not enough. After adding ";" I have also projection issue.

[didier2020]

there is same data (shp format) for only metropole inside this file URL https://static.data.gouv.fr/resources/liaisons-du-reseau-routier-national/20220427-164947/rrn-2022-shp.zip bornage_tout.shp

[didier2020]

thanks ! I'll try with shp (mais sans y arriver ...)

[frodrigo]

Can I take one of your closed PR to fix it ? https://github.com/osm-fr/osmose-backend/pull/1497/files

[didier2020]

not real milestone  

 http://osmose.openstreetmap.fr/fr/issue/a76a49b0-9981-bdd5-45ec-08de5a0fae21  : CS is for "changement de departement"

 http://osmose.openstreetmap.fr/fr/issue/8cf44a2e-238c-b18e-122c-6ecb6e2407b3   : DRG is for "Début Déseau coté Gauche" http://osmose.openstreetmap.fr/fr/issue/0ca6b195-614d-64b4-dcc2-83c3db03b89e  : FRD is for "Fin Réseau coté Droit" real milestone can bee seen on the road are format "xxPRyD" "xxPRyG" "xxPRyU" "xxPRyDC" "xxPRyGC" where xx is a dept number, 

Le vendredi 22 juillet 2022 à 09:07 -0700, Frédéric Rodrigo a écrit :

Can I take one of your closed PR to fix it ? https://github.com/osm-fr/osmose-backend/pull/1497/files — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: <osm-fr/osmose- @.***>

[didier2020]

new one at https://github.com/osm-fr/osmose-backend/pull/1508