We should not have to duplicate postgis spatial indexes

[greenscientist]

The GenericPlaceCollection class create its own spatial index to store points in memory. We should not have to do that kind of operations in Typescript, we have a powerful spatial database optimized for that!

[tahini]

We'll have to list all operations that make use of this local spatial index and move them closer to the database (ie in the backend if they are potentially run in the frontend) so they can benefit from it. Then we can remove this index.

[tahini]

Adding this to the Path refactoring milestone, as it implies revisiting those geographic collection to make them smaller (and thus maybe not sufficient for the required location operations)

[tahini]

One note though, the local indexes are sometimes used to index data that is not yet in the database. For example, when importing GTFS stops, when we aggregate new stops with existing ones, we use this local index, then add new ones or update some after moving their position a little to the centroid of the aggregated nodes.

Only at the end of all node import do we save them in the database. The stop aggregation algorithm will need to be revisited if we want to use postgis.

Proposal:

1. Aggregate locally the new nodes to import first (using either kdbush still, or some function of turf)
2. Aggregate the resulting aggregated GTFS stops with the existing nodes (using postgis)

The advantage of this approach is that is prioritizes aggregating the stops being imported rather than the currently existing nodes, which may have been imported with different aggregation distances and thus may separate two related nodes, which are close by in the current import, but may be aggregated to different original nodes.

[greenscientist]

We might want to revisit the whole agregation concept. Maybe we could save all node data as "raw" node without any agregation. We could then maybe have an addition layer, "NodeAgregated" that we use when we really need to do operation on agreagated nodes.

[tahini]

We might want to revisit the whole agregation concept. Maybe we could save all node data as "raw" node without any agregation. We could then maybe have an addition layer, "NodeAgregated" that we use when we really need to do operation on agreagated nodes.

True, because if you import a stop with a radius of 0, you don't want the next import, with radius of 60, to move your precious node! Which is what is currently happening...

[kaligrafy]

We should update geokdbush and kdbush to new versions until we use only postgis, but it failed after a week of work, because kdbush and geokdbush uses ESM-only export and jest cannot deal with these without a complicated config which I was not able to make work. We could mock the classes that use geokdbush, but it becomes a large rabbit hole so I abandoned...

[kaligrafy]

See this commits which compiled and tested on local (macos) on node 22, but failed on unit tests in github actions (cannot import geokdbush): https://github.com/chairemobilite/transition/pull/1123/commits/4a8343896fa8bd0fc53977430d537c7afa4b1021