path expression to access array elements directly

[koebbingd]

Is there a way to access an array element directly? example:

arr [
  {ip= 192.168.7.1, name="Server 1"},
  {ip= 192.168.7.2, name="Server 2"},
  {ip= 192.168.7.3, name="Server 3"}
]

As far as I understand the library I first have to access the array by calling config.getObjectList("arr") and then iterating over the list to access the fields of my objects. I'd like to have the possibility of calling config.getString("arr[0].ip") or something like that...

Thanks.

[viktorklang]

What's the cost/benefit ratio of this feature proposal?

[havocp]

not currently. you could do getConfigList("arr").get(0).getString("ip") or something though. hardcoding an index in a path string seems like it wouldn't be useful most of the time, so I'm not sure it'd be worth it... the existing way to do things doesn't seem that bad. saving a few characters once in a while vs adding a fair amount more code to support the more complex path expression syntax. don't know.

[viktorklang]

Also, if someone wants to do it they could always create a bridge over to use XPATH anyway. :-1: I think it is something that should live outside of the Config lib as it has nothing to do with configuration.

[koebbingd]

It would be just a convenience function... ...but I see one advantage:

If you have a complex array with many entries as a kind of default and want to override just one value, you could use this syntax too:

arr [
  {a: 1, b: 1, v:1}
  {a: 2, b: 2, v:2}
  {a: 3, b: 3, v:3}
  {a: 4, b: 4, v:4}
....
  {a: 30, b: 30, v:30}
]

"arr[5].v" = 7

Perhaps this use case makes the proposal more interesting?!

[viktorklang]

There's really nothing preventing you from developing such an extension.

[havocp]

Someone else was asking about this lately; but in the context of using it from inside the config file, not in code. I don't know how much more code it would be in the lib, or how often people would use it, though.

[havocp]

If we fix this then #160 could be made to work instead of throwing an exception, if we want.

[ratulmukh]

I think this feature would be really helpful.

I have the same problem that @havocp mentioned (in the context of using it from inside the config file, not in code) . In my case, there exists a base conf file that has an array. I would like my clients to provide their own conf file that would override certain elements of the array in the base conf file. Right now, I have no elegant way to fix it, and that is just too bad. :( Any suggestions?

[havocp]

The best you can do right now is probably to build the array up from ${?foo} references and then people can define (or set to null) each value foo in order to have it in the array or not. They can also prepend or append elements of course.

[tr00per]

Any update on this? It's been 4 years.

[havocp]

Nobody is employed to go through feature requests here. The ones that have been done are the ones that somebody who wanted the feature chose to work on, or chose to hire someone to work on.

If nobody ever wants this enough to work on it or pay for it, then it will never be done. It doesn't matter how many years people have thought it would be nice to have, if none of them thought it was important enough to work on.

Not trying to be grumpy, trying to explain how it works. Most open source projects work the same way.

[ratulmukh]

@havocp, Sorry to say but I mulled over your suggestion to use ${?foo}, but it seemed more like a band-aid and not very elegant, especially given the fact that this can be achieved very nicely if we use JSONPath with JSON files. I was using JSONPath before switching to Typesafe Config, and I had switched for a very strong reason (includes and comments).

My final solution right now is to use Typesafe Config to read and parse the conf files, convert them to strings, feed them into JSONPath, and do the necessary array access operations. There are some disadvantages and inconveniences, but this worked better for me overall.

Your other suggestion of implementing this feature and contributing a pull request is something I might do in the future. Sorry, but I just do not have enough time for it right now.

[movermeyer]

I've run into this same issue when I wanted to check for the existence of a key in a config:

My config has a list of objects with a given attribute. I'm then diff'ing two of these configs and trying to check if a given key exists in a third config.

Example pseudocode in the style of python/pyhocon:

from pyhocon import ConfigFactory
config1 = ConfigFactory.parse_file("file1.conf")
config2 = ConfigFactory.parse_file("file2.conf")
config3 = ConfigFactory.parse_file("file3.conf")

differences = diff(config1, config2)  # diff implemented elsewhere. Returns a list of keys that differ between the two configs. Signature: (ConfigTree, ConfigTree) -> [str]

for key in differences:
    if config3.get(key, None) is None:  # Defaults to None if the key isn't in config3
        raise Exception("Value of {0} differs between config1 and config2, but '{0}' doesn't exist in config3!".format(key))

While diff() can detect the difference in lists within the configs, there is no value for key that diff() can return that will work in this existence check.

I understand that no one is working on this right now, but if someone does, this use case might be interesting.

Currently, I have to extend the key syntax (I did my_list.[0].my_subkey, where 0 is the index within the list), return the extended key syntax from diff() and have special handling for such keys within my calling code. It would be nice to have a formally approved key syntax for referring to the indices of lists.

[ghost]

Ho do I use config.withValue(path, value) without a path string to an array element?

I tried the most natural way you'd think:

my.config.array[0].value

[havocp]

You would need to get the array, manually modify it, and then replace the entire array value

[ghost]

I've been trying this, with config.processes being an array of objects, I did no changes to ml, yet it doesn't work

        val ml = config.getConfigList("config.processes") 
//... change ml
        val c = config.getConfig("config").withValue("processes", ConfigValueFactory.fromIterable(ml))

but it throws:

com.typesafe.config.ConfigException$BugOrBroken: bug in method caller: not valid to create ConfigValue from: Config(SimpleConfigObject({"days":1,"id":"CalendarKlubschule","offsetSec":5,"runEverySec":120,"serviceUrl":"http://localhost:9000/test/klubschule/%s%s%s"}))

at com.typesafe.config.impl.ConfigImpl.fromAnyRef(ConfigImpl.java:281)
at com.typesafe.config.impl.ConfigImpl.fromAnyRef(ConfigImpl.java:273)
at com.typesafe.config.impl.ConfigImpl.fromAnyRef(ConfigImpl.java:194)
at com.typesafe.config.ConfigValueFactory.fromAnyRef(ConfigValueFactory.java:72)
at com.typesafe.config.ConfigValueFactory.fromIterable(ConfigValueFactory.java:114)
at com.typesafe.config.ConfigValueFactory.fromIterable(ConfigValueFactory.java:151)
at Tests.testWebSockets(Tests.kt:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

[havocp]

fromIterable is expecting plain Java objects but it looks like you have ConfigValue objects (or a mix, maybe a plain Java list with ConfigValue in it?) hard to say exactly without the ml-modification code.

[ghost]

Ok managed to put it together. This replaces a global config I access, code is in Kotlin. I use it when, during tests, I want to replace some config vars to make my app testable.

    fun replaceArrayValue(pathToArray:String, key:String, newValue:Any) {
        val cfgList = config.getConfigList(pathToArray)
        val replaceArray = cfgList.map {it.entrySet().associate {
            if(key == it.key)
                Pair(it.key, ConfigValueFactory.fromAnyRef(newValue))
            else
                Pair(it.key, it.value)
        }}

        config = config.withValue(pathToArray, ConfigValueFactory.fromIterable(replaceArray)).withFallback(config)
    }

and this is how you use it:

AdapterConfig.replaceArrayValue("config.processes", "serviceUrl", "newValue")

[vladp]

I am also looking for a way to reference an array element within config. Conceptually, I am trying to simplify configuration and allow users to use 'foreign keys' so to speak into another section. Here is an example:

mainConfig= {

  defaults={
     allowedZones=["zone1,"zone2","zone3"]
  }

     zoneIntroScreen = ${mainConfig.defaults} {
       screensA=[
                { appliesToZone="zone1"
                  attr1="some parm1"
                  attr2="some attr2"},

                { appliesToZone="zone2"
                  attr1="some parm1"
                  attr2="some attr2"},
            ]
     } //end of zone intro
} //end of main config

In the appliesToZone attribute, I would like to reference a particular element of the allowedZones array So that the user, would not make a mistake of typing in "ZONE 1" into appliedToZones values.. as an example.

[WayneWang12]

A simple use case is like:

Say we may have an environment variable or not(most likely it was an ipv6 address). When using IPV6 address, the config of akka hostname shold be [${?IPV6_ADDR}]. And if there is no IPV6, then we should use ${?IPV4_ADDR}.

To aquire a conditional config, I may use this:

real_addresses = [${?IPV6}, ${?IPV4}, "["${?IPV6}"]"]

akka {
  remote {
    artery {
      canonical {
        hostname = "0.0.0.0"      # external (logical) hostname
        hostname = ${?real_addresses.1}      # external (logical) hostname
 }
}

If ${?real_addresses.1} can be resolved, then we can switch smoothly between ipv4 and ipv6 and acheive a way to do conditional config.

So I think it is worthy of implementing such a function. @viktorklang

[viktorklang]

@WayneWang12 Wouldn't you be able to do:

akka {
  remote {
    artery {
      canonical {
        hostname = "0.0.0.0"      # external (logical) hostname
        hostname = ${?IPV4}
        hostname = ${?IPV6}
 }
}

[WayneWang12]

@WayneWang12 Wouldn't you be able to do:

akka {
  remote {
    artery {
      canonical {
        hostname = "0.0.0.0"      # external (logical) hostname
        hostname = ${?IPV4}
        hostname = ${?IPV6}
 }
}

No, because it fails to build up a connnection. IPV6 need to be a [fdbd:dc03:ff:1:1:27:9:14] to be parsed in akka. I may make a pull request to akka to solve this problem.

Play's ws client also need it. Details are here: RFC 2732

Url must be like:

  http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html
  http://[1080:0:0:0:8:800:200C:417A]/index.html
  http://[3ffe:2a00:100:7031::1]
  http://[1080::8:800:200C:417A]/foo
  http://[::192.9.5.5]/ipng
  http://[::FFFF:129.144.52.38]:80/index.html
  http://[2010:836B:4179::836B:4179]

@viktorklang

[voodoo144]

I face the same issue today.

From my point of view HOCON provide this mechanism as "path" for referencing. It should support arrays as well. Notation like ${array[0].array_member_prop} or ${array.0.array_member_prop}

[sd65]

We are facing the same issue today. Our config look like this :

key {
  nested-list = [
    { 
      nested-key = "demo"
    }
  ]
}

We tried to overwrite nested-key like this :

key.nested-list.0.nested-key
key.nested-list[0].nested-key

But as said before it does not work in hocon.

A similar mechanism do exists for java properties and env vars.