3liz
Initial Map loading takes very long
As this thread ist stale and maybe not completely the right topic, i open a new issue
Uff, i thought i was running qgis-server 3.40.4 since quite a while (In Lizmap Admin it told me qgis-server 3.40 is recognized) but two days ago i had to restart my server and since then, the initialisation of my (heavy) project takes ~40 seconds. After the legend symbols are loaded, it is perfectly smooth.
I downgraded to qgis-server 3.34.15 - initialisation takes the same amount of time, but at least the legend symbols are then cached - closing the tab and opening again the same project leads to a loading time of ~6sec.
Should the thread be reopened? Or should i open a new ticket?
Originally posted by @meyerlor in #4750
Since QGIS Server 3.40.X project load in Lizmap takes significant longer. Legend symbols are also not cached anymore, so revisiting a LWC projekt will always download all legend-symbols first and therefore again it will take a lot of time to load the project.
We should look deeper :
- control a cache (client & server side) could be added
- see if another approach could be used to retrieve each layer legend
@meyerlor Do you have a project with data you can share (with enough complexity but non sensitive data) ?
control a cache (client & server side) could be added
@mdouchin sorry for asking again, but there isn't a way to have the json legends pre-generated on the server or client side and avoid LMWC to have to get them generated by qgis server on project load?
@mdouchin sorry for asking again, but there isn't a way to have the json legends pre-generated on the server or client side and avoid LMWC to have to get them generated by qgis server on project load?
It could, but it is never simple to manage synchronization afterwards. For example, the editor saved a modified version of the project, containing new symbology classes, but does not save the Lizmap configuration with the plugin: this would lead to out-of-date list of symbols.
Imagine also some layers with a symbology depending on the variable @lizmap_user or @lizmap_user_groups. Such layers would need to have a different list of classes and symbols depending on the authenticated user.
We will look into it as soon as possible, but we really think this should be managed server side to be efficient and bullet-proof. I know this issue has been raised quite a long time ago, sorry for the lack of feedback.
I had already reported the speed problems on discourse: https://discourse.osgeo.org/t/speed-problems-after-update-to-qgis-server-3-40-4/113474
We have test projects, all on the same type of (very small) cloud server.
Once a quite small project ""Test BürgerGIS..." and once one with several, also non-existent, layers "Test WebGIS-Projekt Bauleitplanung ..." always with Lizmap 3.8.6
Under Apache with QGIS-Server 3.40.5 https://cx21-8.webgis.biz
With Apache and QGIS Server 3.34.15 https://cx22-34-8.webgis.biz (loads significantly faster)
and with nginx and QGIS-Server 3.40.5 https://fossgis.gisgeometer.de/
Another project will follow from @meyerlor Greetings from the FOSSGIS
Hi there, you are pointing QGis-server version but from what I experimented here legend loading latency issues are related to LWC version (>3.6) whichever QGis-server version is used...
It could, but it is never simple to manage synchronization afterwards. For example, the editor saved a modified version of the project, containing new symbology classes, but does not save the Lizmap configuration with the plugin: this would lead to out-of-date list of symbols.
@mdouchin this does not seems a major issue to me, just issue a warning as it is already done when a new layer is added in a project that is already under LM "control".
Imagine also some layers with a symbology depending on the variable @lizmap_user or @lizmap_user_groups. Such layers would need to have a different list of classes and symbols depending on the authenticated user.
I didn't think about it, but seems a corner case. I understand that the right solution must cover all possible cases, but the bottom line is that right now there is an overall sense on slowness affecting any project that is not just 3 or 4 layers, and while it may have got a little better with the latest versions of LMWC and QGIS Server, is still very much an issue affecting many (also by looking at the several threads/comments on this matter here in the tracker).
Happy to chat about this in June in Sweden if you'll be there.
would it maybe be possible, to load the legends after the map layers?
Another option while this issue is not solved is to completely skip the legend and just add a link somewhere to an image of the legend stored in the media folder... What would be the best way to do this ? And how do I display directly an image from the project/media folder in the web browser ? I cannot find direct URL. Thanks for your answers
To remember why we have the JSON legend and how it works in Lizmap.
We have introduce Legend as JSON in Lizmap to display layer simple symbol before the layer name and to provide the capability to check / uncheck categories.
We encountered some issues in the way QGIS Server has implemented JSON Legend. First of all, if we do not do a request per layer, we will not be able to identify the legend of a layer in the response. So server side, Lizmap requested the JSON legend for each layer to return a JSON legend with layer name. The second issue was that QGIS Server does not provide the category key in the JSON Legend to perform check / uncheck categories capability. So we add in Lizmap server plugin a filter to enhance the JSON Legend, but with the limitation of the first point. The category keys only be added for request with one layer.
Now that is said, on the browser side, the legend request is async and the interface does not wait for the response to build the layer tree or the map. I have worked on a way to apply the legend after the tree build.
Has you can see, if the JSON legend failed, the tree and the map are here and has been build without the legend.
All your comments make me think of a possible limitation due to QGIS Server and its installation: do you know the number of QGIS Server processes capable of responding to queries ?
If you only have less than 4 QGIS Server processes to be able to response to request and have 10 layers in your project, I supposed OGC Service capabilities are in Lizmap cache, at least 20 requests have to be treated : 1 per layer for the map, 1 per layer for the legend and all other WFS GetFeatures. All these requests are send in the same time to start the interface and let the server processes the requested queue.
Do you know how many QGIS Server worker are ready to process requests ?
the legend request is async and the interface does not wait for the response to build the layer tree or the map
@rldhont but this is not my observation, and as far as I can see here on the tracker, also is the observation of many others: before having any layer rendered on the map, the legends must be generated. And I'm pretty sure that my setup is ok in what it regards have qgis server set to run in parallel, still the observed response time by a user is frequently very high and it does not get any better at reloading a specific project.
Has you can see, if the JSON legend failed, the tree and the map are here and has been build without the legend
That's why I would like to have it "fail" from the beginning, not to lose one minute of legend loading before displaying the map...
Do you know how many QGIS Server worker are ready to process requests
QGIS_MAP_WORKERS=${QGIS_MAP_WORKERS:-"4"} is what you are asking for ? If so, can I extend it to, like 10 or 12 ?
Do you know how many QGIS Server worker are ready to process requests
QGIS_MAP_WORKERS=${QGIS_MAP_WORKERS:-"4"}is what you are asking for ? If so, can I extend it to, like 10 or 12 ?
Yes it is this variable, but the number of QGIS Server map workers is limited by CPUs / Threads and RAM.
Firstly, no more QGIS Server map workers than the number f CPUs / Threads.
Then QGIS Server uses the RAM to store parsed QGIS Projects, and this cache is not shared. So each QGIS Server will use the same space in RAM. If you have 16 GB in RAM and 12 QGIS Server map workers, each QGIS Server will only have less than 2 Go of RAM that is not enough unless they only have to operate one project. Some projects can take up 1 GB of RAM.
@SecondGIS I have try to load legend and features after layers started to be loaded by the map, and it is not as obvious as it seems. I mean it can't be done in one hour of works.
We can do something but we need at least 1 day of works.
..sooo i offered on several occasions financial support ( via mail) but unfortunately i got no reply. Maybe this is a new chance to renew my pledge, I want to participate and help a sustainable development of lizmap! This is such a great project, @rldhont can you give some guidance on how to support you?
I have try to load legend and features after layers
If this would allow users to start interacting with the map before legend loading is finished it would be perfect.
can you give some guidance on how to support you?
Yes @rldhont I'm willing to participate and waiting to hear from you too on how to proceed
@SecondGIS
QGIS_MAP_WORKERS=${QGIS_MAP_WORKERS:-"4"}is what you are asking for ? If so, can I extend it to, like 10 or 12 ?
Is this a Docker setting for the web server? (Apache?) For me (without Docker) it looks like this for Apache in 000-default.conf: -- FcgidInitialEnv QGIS_SERVER_PARALLEL_RENDERING 1 -- FcgidInitialEnv QGIS_SERVER_MAX_THREADS 8 Server with 8 VCPU 32GB RAM (ccx33 Hetzner CLoud)
This is also used by the QGIS server:
But I still only ever see one process in the above-mentioned project:
I have always set this setting accordingly but never checked its effect. This is also due to the fact that I otherwise provide my own WMS services from other projects. Therefore, there are already several processes.
But might it not work at all? Or at least not in the way I would have expected? Am I doing something wrong or not paying attention to something?
or could there be a QGIS server error? Or can this be optimized on the server?
The cache behavior mentioned by @rldhont does not sound optimal either.
I am on the QGIS-DE e.V. board. We are always looking for useful projects that we can support. The QGIS server is certainly one of them. Last year, for example, we provided support for Complex Features in WFS provider
If there are any ideas for better performance for the QGIS server (and thus also for Lizmap), this would be very interesting.
@SecondGIS I recieved an offer from 3Liz to implement those requests - to organize the funding, can anyone who is interested to chip in some money maybe contact me? meyer (at) gisgeometer.de :) Happy to see that this is moving forward @rldhont !
Yes it is this variable, but the number of QGIS Server map workers is limited by CPUs / Threads and RAM Is this a Docker setting for the web server? (Apache?)
How is this understood in a Docker environment ? How can I determine the optimal setting ?
@SecondGIS I recieved an offer from 3Liz to implement those requests - to organize the funding, can anyone who is interested to chip in some money maybe contact me? meyer (at) gisgeometer.de :) Happy to see that this is moving forward @rldhont !
I would like to hear more from @rldhont as to how his proposal will improve user experience of newer versions of LWC first
those requests
@meyerlor I also would like to have here a clear description of what are the changes that would be implemented.
How is this understood in a Docker environment ? How can I determine the optimal setting ?
independent of Docker, the variables ...PARALLEL_RENDERING and ...MAX_THREADS probably don't do what we thought. The rendering is probably not executed in parallel, but only distributed over several threads, which are processed one after the other (if I have understood it correctly here [1]) This is also my observation, because there is always only one active process. I don't know whether the variables from the link improve anything, but I'm afraid not.
[1] https://discourse.osgeo.org/t/geschwindigkeitsprobleme-mit-qgis-server-3-40-x/146307/2
@SecondGIS @gioman we made two proposals.
The first one is
Delaying GetLegendGraphic and GetFeature requests after GetMaps
We propose to shift the generation of GetLegendGraphics and GetFeature requests to after the map has been created and all layers added to it, so that GetLegendGraphics and GetFeature requests leave the client after the GetMap requests generated by OpenLayers and arrive at QGIS Server after the GetMap requests. This would be the same as before 3.7.
This modification will be applied to version 3.8 and subsequent versions.
The second one is
Delaying GetLegendGraphic and GetFeature requests after GetMaps
We propose to shift the generation of GetLegendGraphics and GetFeature requests to after the map has been created and all layers added to it, so that GetLegendGraphics and GetFeature requests leave the client after the GetMap requests generated by OpenLayers and arrive at QGIS Server after the GetMap requests. This would be the same as before 3.7.
We also propose to replace the GetLegendGraphics POST request with a GET request per layer, adding HTTP headers to take advantage of the Web cache.
These modifications will be applied to version 3.8 and subsequent versions.
I don't know if this changes will really enhance the user experience, but with it, the map will be rendered before the legend and other tools that requested features (location by layer, form filter, etc).
In my tests, I have discovered that OpenLayers is slower than the rest of our code.
@SecondGIS and @guenterw QGIS_MAP_WORKERS is a docker variable for Py-QGIS-Server. It is not available for Apache and FCGI.
The variables PARALLEL_RENDERING and MAX_THREADS is not for managing the number of QGIS Server Map processes available to response to requests. These variables is also available for Py-QGIS-Server and manages the way images (GetMap) and maps in tempalte (GetPrint) are build by QGIS Server. The variables is usefull when multiple layers have to be rendered by QGIS Server. The QGIS renderer draw each layer and layer's label before merged them it the image. When PARALLEL_RENDERING is activated with '1', 'true' or 'on', the layer drawing and label is performed in parallel with the number of threads defined by MAX_THREADS varaible before merged them in the image.
I don't know how to manage the number of QGIS Server map processes with Apache and FCGI. We have build Py-QGIS-Server to manage the number of QGIS Server map processes and to keep them alive to do not loss the QGIS Server Projects cache.
@rldhont thank you for the clarification. Is there a noticeable improvement in speed when using Py-QGIS-Server? Is it therefore advisable to take a look at Py-QGIS-Server or to use it?
I have found this, for example: https://github.com/3liz/py-qgis-server/issues/26 But I'm not sure whether it's worth the effort to test this and whether my knowledge is sufficient.
Is there a noticeable improvement in speed when using Py-QGIS-Server? Is it therefore advisable to take a look at Py-QGIS-Server or to use it?
@guenterw @rldhont +1 on this question. Certainly is my fault that I never looked into it as I should have, but I was under the assumption that in production was not really an alternative to apache or ngnix.
The variables is usefull when multiple layers have to be rendered by QGIS Server. The QGIS renderer draw each layer and layer's label before merged them it the image. When PARALLEL_RENDERING is activated with '1', 'true' or 'on', the layer drawing and label is performed in parallel with the number of threads defined by
MAX_THREADSvaraible before merged them in the image.
@rldhont so... if I understand correctly qgis server works in parallel when it has to render to image a specific layer that may be defined by several parts (the map itself, labels, diagrams, etc.), but if more than one layer is requested then those requests always work in series?