Very slow performance with large number of env vars represented as nested config

[notxcain]

When you run your app on Kubernetes there are a lot of env vars defined. At least two for each service. So in my case there are 800 of env vars like LONG_SERVICE_NAME_HOST and ANOTHER_LONG_SERVICE_NAME_PORT.

And it seems that config loading is stuck forever in PropertyTree#mergeAll.

[afsalthaj]

@notxcain thanks for reporting the issue. I will try to reproduce this and take a look at this issue.

[amitksingh1490]

+1

[tusharmath]

For our use-case (100kb of env variables) we get an out of memory error —

java.lang.OutOfMemoryError: Java heap space
	at scala.collection.immutable.BitmapIndexedMapNode.copyAndSetNode(HashMap.scala:937)
	at scala.collection.immutable.BitmapIndexedMapNode.updated(HashMap.scala:730)
	at scala.collection.immutable.HashMap.updated(HashMap.scala:152)
	at scala.collection.immutable.HashMap.updated(HashMap.scala:39)
	at scala.collection.immutable.MapOps.$plus(Map.scala:133)
	at scala.collection.immutable.MapOps.$plus$(Map.scala:133)
	at scala.collection.immutable.AbstractMap.$plus(Map.scala:641)
	at zio.config.PropertyTree.$anonfun$merge$6(PropertyTree.scala:135)
	at zio.config.PropertyTree$$Lambda$547/0x0000000100330040.apply(Unknown Source)
	at scala.collection.immutable.List.map(List.scala:246)
	at zio.config.PropertyTree.$anonfun$merge$1(PropertyTree.scala:135)
	at zio.config.PropertyTree$$Lambda$543/0x000000010032d040.apply(Unknown Source)
	at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:638)
	at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:634)
	at scala.collection.AbstractIterable.foldLeft(Iterable.scala:920)
	at zio.config.PropertyTree.merge(PropertyTree.scala:128)
	at zio.config.PropertyTree.merge$(PropertyTree.scala:123)
	at zio.config.PropertyTree$Record.merge(PropertyTree.scala:223)
	at zio.config.PropertyTree$.$anonfun$mergeAll$2(PropertyTree.scala:277)
	at zio.config.PropertyTree$$$Lambda$541/0x000000010032b840.apply(Unknown Source)
	at scala.collection.immutable.List.flatMap(List.scala:293)
	at zio.config.PropertyTree$.$anonfun$mergeAll$1(PropertyTree.scala:277)
	at zio.config.PropertyTree$$$Lambda$539/0x000000010032a840.apply(Unknown Source)
	at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:168)
	at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:164)
	at scala.collection.immutable.List.foldLeft(List.scala:79)
	at zio.config.PropertyTree$.mergeAll(PropertyTree.scala:276)
	at zio.config.PropertyTree$.unflatten(PropertyTree.scala:271)
	at zio.config.ConfigSourceStringModule$ConfigSource$.fromMapInternal(ConfigSourceModule.scala:439)
	at zio.config.ConfigSourceStringModule$ConfigSource$.fromMap(ConfigSourceModule.scala:226)
	at zio.config.ConfigSourceStringModule$ConfigSource$.$anonfun$fromSystemEnv$3(ConfigSourceModule.scala:386)
	at zio.config.ConfigSourceStringModule$ConfigSource$$$Lambda$124/0x00000001001b5840.apply(Unknown Source)
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects

[afsalthaj]

@notxcain Could you also give the details of the source (Example: is it key -> value property file or HOCON) ? What's the target data structure (Example: case class, Map?)

For example, this works:

  val map =
    (0 to 5000).toList.map(r => s"LONG_SERVICE_NAME_HOST_${r}" -> "value").toMap

  val getHost = string("LONG_SERVICE_NAME_HOST_5000")

  println(read(getHost from ConfigSource.fromMap(map)))

[notxcain]

@afsalthaj I've sanitized our env vars https://gist.github.com/notxcain/c21f24e2d1f8e8e692512acdb496c599.

[afsalthaj]

Sure. Could you also share the code that tries to retrieve the config?

On Wed, 28 Oct 2020 at 8:59 PM, Denis Mikhaylov [email protected] wrote:

@afsalthaj https://github.com/afsalthaj I've sanitized our env vars https://gist.github.com/notxcain/c21f24e2d1f8e8e692512acdb496c599.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zio/zio-config/issues/418#issuecomment-717825572, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABY2QJIQSQQJMCBVEMQWXK3SM7TPXANCNFSM4SPD6MIA .

[notxcain]

@afsalthaj just use this map in your example.

[afsalthaj]

@notxcain Worked

[notxcain]

Damn, I must've oversanitized something. It fails on unsanitized env vars. Let me recheck.

[afsalthaj]

@notxcain Sure, thanks. Meanwhile I am looking at @tusharmath's example that has a much more complicated retrieval by passing delimiter to 100KB key value pair.

[afsalthaj]

@notxcain See if you are also passing delimiters. Example: val mapSource = ConfigSource.fromMap(source, keyDelimiter = Some('_')) Then we all are on the same page.

[notxcain]

Ah, of course! Sorry I didn't mention that.

[notxcain]

@afsalthaj so now you see it too?

[afsalthaj]

Yes, only when passing the delimiters. That's how I reproduced the issue. Fixing mergeAll function now

[notxcain]

@afsalthaj how is it going? Did you find the solution?

[notxcain]

@afsalthaj sorry to bother you, but do you have an update?

[afsalthaj]

@notxcain could I get back to you in a while. For the time being if you filter the values first in sys.env and then pass it as a reduced map to zio-config for the time being, it will fix the issue. The issue is when converting a 800 plus flattened values as nested structure using delimiters.

[afsalthaj]

@notxcain give me a bit more time, and I hope il fix it soon. Currently busy with Api docs and zio-columnar

[notxcain]

As a workaround, we used system properties populated from env vars.

[notxcain]

Have you thought about lazy source design? So that you don't have to convert all the env vars, and just access on demand.

[afsalthaj]

@notxcain Sorry to get back late. I was really busy with some of the other changes.

The memory issue comes into picture when there are large number of environment variables in the source with nested behaviour, that is, when passing explicit passing for delimiters.

The quick fix for you is not to pass delimiters if there are too many delimited key-value pairs in flattened sources. We will be looking at a proper solution than jumping on to a quick hack :)

Thanks for the patience.

[jdegoes]

A mutable implementation of mergeAll would fix this, as would, potentially, making source more lazy. I think I'd prefer the former as if you have thousands of env variables inside a property tree, laziness won't help.

Anyone want to volunteer for a mutable version of mergeAll? A good exercise in high-performance functional Scala.

[tusharmath]

@jdegoes That would be me ✋

[jdegoes]

@tusharmath Awesome! Let me know if you would like to pair on it or need additional clarification! 🙏

[remiguittaut]

Hi guys, I see that there is a standing PR. That problem is a bit of a blocker for me. Any idea about when this can be deployed?

[afsalthaj]

There is a slight redesign going on at the configsource side. We have an eye on this.