Config: more than 10 configurations found

[hokascha]

I have more than 10 tileset configurations. Renderd refuses to start with: Mar 01 14:34:16 maps renderd[24134]: Config: more than 10 configurations found

Is that limit configurable somehow? Why was it made? Can I circumvent the limit?

Thanks!

[hummeltech]

If you aren't compiling mod_tile yourself, there's not much you can do at this time. If you are, however, you can adjust the value of XMLCONFIGS_MAX before compiling.

Here is where the value is defined: https://github.com/openstreetmap/mod_tile/blob/04a0bf0f0985bfe568001b0de4520468bdc971e2/includes/render_config.h#L61

There has been some interest and effort in increasing the default value, but it has not yet been implemented.

[hokascha]

Thank you, I have adjusted the value and recompiled. This works up to a value of 490. Anything above that leads to a SegFault. I suspect that somewhere deeper in the code, XMLCONFIGS_MAX is used for the initialization of other objects and hits limits there.

[hummeltech]

I suspect it is because there are XMLCONFIGS_MAX instances of xmlmapconfig being created for every render_thread

https://github.com/openstreetmap/mod_tile/blob/7389b160bf6af3c50272e369dd083717ad0c473b/src/gen_tile.cpp#L367-L370

This is likely exhausting the thread's maximum stack size limit, which can be found like so:

$ ulimit --stack-size

On my system the limit is 8192 (I.E. 8388608 bytes) which is exceeded when setting XMLCONFIGS_MAX to 500 as can be seen in this backtrace:

#0  0x00006250cc6fafa9 in render_thread (arg=<error reading variable: Cannot access memory at address 0x78fc55bdc338>) at ./src/gen_tile.cpp:368
        parentxmlconfig = <error reading variable parentxmlconfig (Cannot access memory at address 0x78fc55bdc358)>
        maps = <error reading variable maps (value requires 8524000 bytes, which is more than max-value-size)>
        i = <error reading variable i (Cannot access memory at address 0x78fc55bdc344)>
        iMaxConfigs = <error reading variable iMaxConfigs (Cannot access memory at address 0x78fc55bdc348)>
        render_time = <error reading variable render_time (Cannot access memory at address 0x78fc55bdc34c)>
#1  0x000078fc636a955a in start_thread (arg=<optimized out>) at pthread_create.c:447
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {133025174128320, 6261848273327184998, -2232, 25, 140720486476208, 133025165737984, -6404644596209460122, -6404599127370277786}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 
              0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#2  0x000078fc63726a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[SomeoneElseOSM]

I can see why someone might want more than 10 configs (I'm nearly up to that), but is 490 ever going to be a realistic number that someone might want to set?

[hummeltech]

Yes, I agree, I think that if someone really needs to have so many configs, they might at the very least want to consider a different deployment method.

There is a pull request which raises the limit to 100, which is likely a fair value, especially with the other changes included along with the increase which will also help to decrease the stack usage. #293

[hokascha]

For example, I am visualizing sensor data as some kind of heatmap. That heatmap is generated once per day and the user can freely enable overlay layers for any date - resulting in a huge amount of different configs. However, this would also be possible using vector layers at the cost of performance.

A second possible use case would be something like mapbox where users can create their own map styles and get an own config for every user map.

In the end, there will always be memory limits in the current architecture. We would need some kind of dynamic configuration to solve this. In my case, it's simply the underlying source shape file that changes, so having variables in the config like $URL (even allowing wildcard urls) and passing that to the XML interpreter could easily provide hundreds of different configurations in a single renderd config section. But I am not deep enough in the code to estimate if this is possible.

[hummeltech]

Alright, that makes sense. For the heat map usage, I might suggest looking at writing a "parameterization" function to handle the variable date. The tiles would then be accessible under http://{host}/{uri}/{date}/{z}/{x}/{y}.{extension}

There currently exists only a function for language, but it shouldn't be too difficult to do the same for a date, here is that code: https://github.com/openstreetmap/mod_tile/blob/7389b160bf6af3c50272e369dd083717ad0c473b/src/parameterize_style.cpp#L29-L79

And here's a bit of background: https://lists.openstreetmap.org/pipermail/dev/2013-October/027472.html

Also, I have been working on various maintenance tasks (including freeing up some stack space), which might allow enough configs for your second use-case. https://github.com/openstreetmap/mod_tile/compare/master...hummeltech:RenderThreadStackUsage?w=1

But you are correct, as it is currently, there will always be a hard limit.

[bkastner]

I'll add my vote to this, I'm in a similar situation to the OP. Agreed that a limit of 100 seems reasonable.