script
script copied to clipboard
bitfield
Add "raw" flag equivalent for JQ
Hello! Thank you for the amazing work :)
I would like to suggest the possibility to output a string using JQ but with the equivalent of "-r" or "--raw" we can find on the official jq command: this allows to get a clean output for a single field for example, instead of having to sanitize it by doing some triming.
What do you think?
Thanks!
Nice suggestion, @LorisFriedel! Can you give a brief example of a program like this, showing how you'd like this behaviour to work?
For example, I would ideally do this to find a single element in a json object and use it directly:
nodeName, err := script.
Exec("kubectl get pod myPod -o json").
JQRaw(".spec.nodeName").
String()
// output would be: xxx-ffkch-something-e32as-someregion-gsbn4
Today, that's what I need to do:
nodeName, err := script.
Exec("kubectl get pod myPod -o json").
JQ(".spec.nodeName").
String()
// output is: "xxx-ffkch-something-e32as-someregion-gsbn4"\n
nodeName = strings.Trim(strings.TrimSpace(nodeName), "\"") // extra step to sanitze the output
// output is then: xxx-ffkch-something-e32as-someregion-gsbn4
In fact the trailing "\n" is almost always a bit of a pain to handle, not really related to JQ :/
Maybe this could be done with a more general-purpose trim method—something like Trim?
nodeName, err := script.
Exec("kubectl get pod myPod -o json").
JQ(".spec.nodeName").
Trim().
String()
// output would be: xxx-ffkch-something-e32as-someregion-gsbn4
This could be, but the trim would need to strip spaces AND quotes, and that would make it a weird "Trim" method I guess, WDYT?
Maybe just a Trim(cutset string) and TrimSpace() functions could be perfectly suited here, easy to implement and easy to understand as they match their strings.Trim and strings.TrimSpace equivalent! WDYT?
I'd like to gather a bit more data on exactly how people are using JQ within script programs, so that we can come up with an API which fits well in that context. So if you're reading this issue and you have a program that uses jq queries, please post it in a comment!
Hi @bitfield,
I'd like to gather a bit more data on exactly how people are using
JQwithinscriptprograms, so that we can come up with an API which fits well in that context. So if you're reading this issue and you have a program that usesjqqueries, please post it in a comment!
I cannot speak for every jq user out there however, using jq with the -r command-line switch is definitely a thing, as otherwise, you often have to resort to some sed, grep or awk invocation to remove the unwanted quoting and/or newlines.
Update:
That said, given your proposal in the comments https://github.com/bitfield/script/issues/125#issuecomment-1158663852 and https://github.com/bitfield/script/issues/125#issuecomment-1175220675, perhaps we could add Trim(), we could also add good 'ol Tr() (man 1 tr) to keep it close to the shell scripting as much as possible.
using jq with the -r command-line switch is definitely a thing
I believe you, but we need a realistic example to use as the input to the design process.
Hi @bitfield,
using jq with the -r command-line switch is definitely a thing
I believe you, but we need a realistic example to use as the input to the design process.
Thinking about it some more... I wouldn't do it.
Adding some convenience functions like Trim() or Strip(), and also Tr() might be a better alternative - new API could be easily reused somewhere else, other than removing bloat from jq's output.
Anything like Trim(), Strip() and such would save a lot a painful code handling those polluted outputs indeed!
Is there any example you can think of where a 'trim spaces and quotes' method would be used with anything other than JQ?
For the "trim quote" part, not really, but for the "trim space" part I guess everywhere where Fprintln() function is used in the library could be a good candidate, as when getting the output of something you generally don't want additional whitespace that serves no purposes!
Hi @bitfield,
Is there any example you can think of where a 'trim spaces and quotes' method would be used with anything other than
JQ?
I think, if anything, we would want the following:
- Trim() (or Strip(); whichever naming works best)
- Tr(set1, set2)
The Trim() can be used as part of the pipeline and such to sanitise whitespaces - both leading and trailing at once. Perhaps a specialised variety could be added too if it at all makes sense, so that would be TripLeading() and TrimTrailing(), but I am not sure if we need this (an open question here).
An example use case (aside from using it for JQ): working with a result of a process spawned with Exec() to clean up output from the external process.
The Tr(), that could be used to strip anything... anything you see fit from the pipeline.
For example:
script.Exec(`echo 'a" b`).Tr(`" `, "").Stdout()
Would produce:
ab
This would be akin to running tr -d where leaving the second argument empty would mean we want to remove the set that matches rather than replace it with something as far as an API idea goes.
An example use case (aside from using it for JQ): working with a result of a process spawned with Exec() to clean up output from the external process.
The problem I see is that a Trim method specialised for trimming jq output most likely wouldn't be much use for anything else.
Conversely, a Trim method that's configurable enough to trim both jq data and a variety of other cases would probably require an inconvenient amount of paperwork.
It's possible that we could get the best of both worlds, by providing a Trim method that performs general-purpose substitution, like tr, and also a "raw mode" for JQ that could use Trim under the hood.
Hi @bitfield,
An example use case (aside from using it for JQ): working with a result of a process spawned with Exec() to clean up output from the external process.
The problem I see is that a
Trimmethod specialised for trimmingjqoutput most likely wouldn't be much use for anything else.
Not if we make it generic. I can see it being useful as a handy function that can be used to rid something in the pipeline, so to speak, of whitespaces and such.
Conversely, a
Trimmethod that's configurable enough to trim bothjqdata and a variety of other cases would probably require an inconvenient amount of paperwork.
What if we kept it simple? Like String#strip as an example from Ruby.
It's possible that we could get the best of both worlds by providing a
Trimmethod that performs general-purpose substitution, liketr, and also a "raw mode" forJQthat could useTrimunder the hood.
Or... Tr() and Strip(), simply.
Users of JQ can then use either depending on their needs. Unless you do want to have JQ() and JQRaw().
Thoughts?
Well, it's not just whitespace that needs to be removed from the JQ data; it's quotes, too. A Strip method that worked like Ruby's String#strip wouldn't do that, I suppose. So we couldn't write this:
script.Echo(data).JQ(query).Strip().Stdout()
On the other hand, suppose there were some method Translate (Tr means nothing unless you already know Unix), that worked like tr, we'd have to write:
script.Echo(data).JQ(query).Translate(" \"", "").Stdout()
...which is a lot to have to write after every call to JQ.
As I say, cleaning up JQ data is such a specific task that I can't see a no-argument method to do it being useful for anything else.
Hi @bitfield,
Well, it's not just whitespace that needs to be removed from the JQ data; it's quotes, too. A
Stripmethod that worked like Ruby'sString#stripwouldn't do that, I suppose. So we couldn't write this:script.Echo(data).JQ(query).Strip().Stdout()
The idea to add Strip() was not strictly related to JQ(), but more as a generic function (a helper, if you wish) that could be used here and elsewhere to remove leading and trailing whitespaces.
On the other hand, suppose there were some method
Translate(Trmeans nothing unless you already know Unix), that worked liketr, we'd have to write:script.Echo(data).JQ(query).Translate(" \"", "").Stdout()...which is a lot to have to write after every call to
JQ.
I see your point. OK. To keep it simple and retain the same ergonomics as other functions, introducing JQRaw() next to JQ() might be a good idea then.
As I say, cleaning up JQ data is such a specific task that I can't see a no-argument method to do it being useful for anything else.
The Strip() and Translate() were intended to be as generic as possible so that both could find use in different places too.
This could be, but the trim would need to strip spaces AND quotes, and that would make it a weird "Trim" method I guess, WDYT?
The GoLang string.Trim routine takes a cutlist which can trim multiple different characters. I don't think it would be weird at all.
Right, but if we had (for example) JQRaw, or equivalent, would we really need a general-purpose Trim method? What for?
I would use Trim to remove unwanted characters either from the start, the end, or both ends of a string.
Yes, understood, but why would you need to do that? In what real-world application would you need to trim specific characters from either end of a string? Don't take this as a challenge—I'm not saying there's no use case for this—but as an invitation to contribute suggestions for ways this feature might be used.
If a number of intelligent, experienced people cannot, in practice, think of any situation in which they would really use this feature, then I'm happy to leave the feature out. The best tools tend to be the ones that aren't bloated with unnecessary features.
Thanks @ John for your great work on script. I am also in the same kind of questions like you guys. On my side I am regularly using jq to quickly strip some json payload. In my case, I am using jq -r to use the raw possibility (no quotes around attributes). For example : echo '{ "ip": "192.168.0.2"}' | jq -r .ip will give 192.168.0.2. There are a lot of args with the jq and almost the same with the gojq that is in use in JQ -- see https://github.com/itchyny/gojq/blob/main/_gojq. I try to find a workaround to pass arguments to the JQ implementation (but I didn't succeed) or to define arguments thru environment variables like SCRIPT_JQ_PARAM="-r" . Did you try this idea ? Or to modify the JQ() method to accept arguments like : JQ("-r", '{ "ip": "192.168.0.2"}') ?
I suspect a simple JQr would be the most straightforward API.
script.Echo(`{ "ip": "192.168.0.2"}`).JQr(".ip").Stdout()
// Output:
// 192.168.0.2
What do you think?
Yes but you will add a new function to your script list. I am not sure this is what you want ... Also as I mentioned there are many other parameters that I use like the S (slurp) or the -C for instance. But again, after reading the argument list in gojq, this may be not a good idea to get access to some parameters. So yes why not JQr or JQraw ... Simple and easy. However I also like the idea to have a kinda generic "translator" in your set of functions. Thanks anyway.
It doesn't look like gojq has an easy way to get "raw" results, so now I'm leaning towards something like Trim for cleaning up the output.
Hello John, I have also the idea to have a generic function in the pipeline of functions. For example : script.File(file).JQ(".ip .host" ).Func(myTrimFunction).String(). In that case, we read a JSON file, we get the ip and host from each line with the double quotes and we apply a processing function (in that case myTrimFunction) and finally get a string at the end. MyTrimFunction is my customized routine to do the transformation job ... What do you think ? The advantage is we have our own personal routine that could be trimming double quotes or converting or translating any character that we need.What do you think ?
You can do exactly this with FilterLine (for example):
script.File(path).JQ(".ip .host").FilterLine(func (s string) string {
return strings.Trim(s, `"`)
}.String()